DOCUMENT RESUME 

RC 014 528 

Bhola, H. S* 

Evaluating Development Training Programs* (A 

Practical Manual for Trainers of Development Workers 

and Change Agents in the Third Wo: Id). 

German Foundation for International Development, Bonn 

(West Germany) * 

82 

293p* 

Guides - General (050) — Books (010) 
MF01/PC12 Plus Postage* 

♦Change Agents; Community Development; Data Analysis; 
Data Collection; Definitions; Developing Nations; 
Evaluation Criteria; *Evaluation Methods; Foreign 
Countries ; * Improvement programs; Mater ial 
Development; Models; professional Training; * program 
Evaluation; Rural Development; * Staff Development; 
Technical Writing; M'raining Methods; Training 
Objectives 

♦Development Education 



A manual for trainers of development workers and 
change agents in the third world provides guidelines for evaluation 
of development training programs* Chapters define evaluation, present 
a geneial model of development, and discuss evaluation in development 
training programs. Designing training systems for development workers 
and change agents, asking evaluation questions, making evaluation 
agendas, writing formal proposals for evaluation studies, and 
monitoring and quick appraisals are covered in subsequent chapters. 
Naturalistic and scientific paradigms of evaluation, concept 
analysis, changa indicators, quality standards, and evaluation tools 
and instruments are discussed* Other chapters detail product 
evaluation \pre~testing of instructional and training materials) , 
designing evaluation £ :udies, and the concept of design in 
naturalistic evaluation* Implementing evaluation studies is discussed 
in connection with problems of data collection in the real world* 
processes and techniques of data analysis and statistics for 
e valuators of development training programs are covered * Guidelines 
for writing evaluation reports are given* A final chapter discusses 
politics of evaluation and evaluation standards* An appendix covers 
history and use of the Action Training Model* A bibliography presents 
sections on training design (26 entries), evaluation models and 
techniques (65), and evaluation of training and development (31)* A 
glossary defines relevant terms* (MH) 



ED 238 651 

AUTHOR 
TITLE 



INSTITUTION 

PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 



*********************************************************************** 

* Reproductions supplied by EDRS are the be s t that can be mad ft * 

* from the original document* * 
*********************************************************************** 



ERIC 



EVALUATING DEVELOPMENT TRAINING PROGRAMS 
(A Practical Manual for Trainers of Development 
Workers and Change Agents ffl thf* Third World) 



By 

». S. Bhola 
Indiana UniversUv 



OCUTSCHF ST If TUNG FUR t«Tf RNATtOWALF rNTWlCKlNltn 
German Foundation for International Development 



To my wife, JogitHier — 
One* toutjh ^valuator! 



2 

ERIC 4 



TABLE OF CONTENTS 

Page 

Preface 5 

CHAPTER I: What is evaluation? IS 

11: A general model of development 72 

Ml: The promise of evaluation in development training 

programs 21 

IV: Designing training systems for de/eloPment workers 

and change agents 3? 

V: Asking evaluation Questions, making evaluation 

agendas, and typical evaluation themes 54 

VI: Writing formal proposals for evaluation studies .... 71 

V1 1 : Monitoring and quick appraisals 87 

VIII: Models of evaluatloo 93 

IX: Concept analysis, indicators of change and 

standards of Quality MS 

X: Evaluation tools and Instruments 145 

xl: Product evaluation or pre-testing of Instructional 

and training materials 

Xl 1 s Designing evaluation studies, and the concept of 

design in naturalistic evaluatloo 177 

3 

ERLC ° 



xl 1 1 : ijiiplettenting evaluation studies: problems of 

data collection in the real world 193 

XIV: The process and techniques of data analysis 193 

XV: ttasic statistics for ^valuators of deve lopntent 

training Programs £26 

XVI: Writing evaluation reports 242 

xvff: Politics of evaluation and evaluation standards .... 253 

APPHlDIX I: The Action Training Model 2*2 

Bibliography " , . . . 272 

Glossary of terms , . . . 2U2 



4 

ERIC e 



PREFACE 




Ibis Is a book on" the evaluation of develnPment training Prog*,-** 
courses* camPs* seminars and workshops organized to train development 
workers , 

Development is on the national agendas of almcU all of the Third 
World countries today, lo teach farmers, workers and housewives i« 

r 

villages and cities, new knowledge* new attitudes and new skills, 
hundreds of thousands of development workers (or change agents) are 
being trained all over the Third World. 

Thfs training Is being conducted through ad (wc_ training programs; 
in training centers and Institutes specially set up for the purpose; or 
in universities and other Institutions of higher education. Training 
objectives vary from the economic to the social and the political. 
Training philosophies and methodologies differ widely. The levels «f 
personnel 'trained range from front-line workers to specialists. Planners 
and managers at district, regional and central levels, Colli civil 
servants and volunteers may be covered. Training experience; may v,iry 
1" duration from half a day to a couPte of years. 

Accumulated experiences fn the training of change agent* varies, of 
course, from country to country, Sonne developing countries have 
aheady built extensive networks of Institutions to meet tbe training 
needs nf development workers, India In Asia, Kenya In Afrlra and ftraril 
in South America could be cited examples of such countries. Other 
countries have not been as fortunate. They have h<>en constrained bv lack 
of resources and still have a long way to go in fulfilling their develop- 
ment training needs, 
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A mood of stocktaking of training experiences, however, seems today 
to be widespread. Countries that already have gained long experiences 
in th\: training of change agents want to evaluate their experiences. 
Others who are just beginning* want to make $ y re that they dn not 
repeat the mistakes that the pioneers may have made before them. Within 
the development training sector* those currently engaged iri development 
training as trainers or as administrators wish to be able to evaluate 
training objectives and training methods and materials* to maximize 
iutojct on their trainees* and ultimately on the communities undergoing 
development. Evaluation of the effectiveness of development training 

4 

Is* therefore* today an area that is both quite popular and inherently 
signii leant. ^ - m 

Thib book K dddr'es^ed to the trainers of development workers and 
change agents for various development sectors, such as adult basic 
education and adult literacy, agricultural extension, family Planning* 
health arid nutrition education, cooperative education* community 
development* social welfare* development administration, leadership 
training, development support communication* integrated rural development, 
trade union education ami political education. Focus is on the Third 
World the areas of poverty in Africa. Asia and Latin America. This 
particular choice of audience, their educational background and 
experience* and the conditions of iheir work have determined the content, 
the general approach and the style of this book. While development 
trainers are the primary jutfience of this book* those engaged in the 
planning and administration of development programs in national and 
international organizations might also find tne book of some interest* 
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The five Introductory chapters Provide a framework for the discussion 
of amthotls and' techniques of evaluation included *n later chapters. The 
first chanter of the hoofc states the essential Purpose of conducting 
evaluation* fn a second chapter* a general model of development Is 
presented. The *odel points up the significant role of human resource 
development within the overall developmental processes and puts the need 
for training development workers and change agents in the proper perspective* 
The returns that will roost likely accrue from the evaluation of development 
training Programs are then weighed and considered* 

fn the fourth chapter of the book, some preliminary ideas on 
training design are offered, A model for developing useful descriptions 
of various training Programs Is t ten included* on the Premise that to . 
evaluate a Program, the evaluator must first be able to desert he It in 
conceptual terms. This chapter should enable readers to inok at training 
Programs as training systems, with particular structures and Processes 
within, and Particular Interfaces and Interconnections without. The 
model Presented In this chapter may also provide norms and standards for 
making evaluative judgments In regard to training programs being 
evaluated. Finally, In the fifth Introductory chapter evaluation 
Questions of significant interest to development .trainers are generated! 
and the criteria for the choice of training Issues for evaluation aro 
discussed* 

In the chapters that follow (Chapters VT to XVI11)» the whole array 
of evaluation topics ba< been discussed Including writing evaluation 
proposals, monitoring and quick appraisals, evaluation models* cnncoPt 
analysis, development of Indicators* evaluation desion* construction of 



tools and instruments, Product evaluation, data collection and data 
aii^lys is * basic statistics and dissemination of evaluation results, 

all tins content has been included in this one book because all these 
loPitb do go together. Also* because this may be the only book our 
readeis may have access to or may read for some tiwe. The book presents 
what we hope is a Judicious mix of the theoretical ami the practical 
content. Topics such as the development model* training design and 
evaluation models > by necessity, were discussed at a highly conceptual 
level. All other topics have been discussed at a much more practical 
level- 

Lva luatj ny Dev e lopment Training Programs is meant to be a first 
systuuatic introduction to the evaluation of development training programs. 
This 15 not a book for the practicing evaluator* nor do we seek to prepare 
evaluation experts . Our aspirations are more modest. We seek to prepare 
"tranters-evaluators" professional trainers who have acquired a 
sufficient level of evaluation skills to be able to develop needed feed- 
hack for the improvement of their development training programs. 

Hie emphasis is on internal evaluation; evaluation handled by the 
traWr and often relating to the trainer's own progiam. We promote low- 
cost evaluation and suggest that evaluation of dettslopnent training be 
handled as far as possible from within the resources already available 
to the training program or to the training institution. 

uoth formative evaluation (evaluation conducted during the formation 
of the program with the Intent to improve us des1gn)i and sumiatlve 
evaluation (evaluation conducted at the end of a program to sum up its 
resuJts) have been covered. Both the scientific (quantitative) and the 





naturalistic (qualitative) paradigms have been discussed. Our 
methodological orientation Is pragmatic* even opportunistic. We believe 
that seme feedback on training effectiveness Is better than none at air, 
and If the best of evaluation design and measurement Is not possible In 
a situation* the second best Is all right too. 

The language used In the book has been as simple as possible, where 
technical terms had to be used, they have been explained i n the text* 
and more elaborate definitions have been Included In the Glossary at the 
end of the book. 

The book does not promise miracles* however. It wHl have to be 
read and read carefully by trainers-evaluators. Seme chapters and 
Portions of other chapters may need to be read more than once for maxtmim 
benefit. References to the dictionary or to the Glossary of terms may be 
necessary. Suggested applications of seme of the concepts will need to 
be actually worked out; and statistical tests must be learned by applying 
them to available data. In seme cases* additional reading will be railed* , 
for. 

When used as a text for seminars and workshops of short duratlnn, 
the various chapters of the book need not be read In the order in whitfi 
they have been presented in the book. Both the content and the 
organization of content for the workshop should be developed partlclpatlvely* 
with the particular workshop group, In terms of the particular needs and 
interests of tha* group* at that Particular time. Thus, one workshop may 
start with now to develop an evaluation proposal and go through writing 
evaluation questions* to developing Indicators* to constructing question- 
naires and structured Interview schedules, to the analysis of data. 
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Another workshop may sUrt with descriptions of training systems and go 
through developing indicators^ to developing questionnaires to bask 
statistics, Vet another workshop could start with the models of 
evaluation and* in the choice of other content* way emphasize naturalistic 
evaluation strategies. 

in each case* workshop organizers should point out clearly to 
participants what wii1 be read from the text* in what sequence, by what 
Lime* indicating page nunbers for each assignment. Participants should* 
at tne same time* be encouraged to read the whole book from the beginning 
to the end. This should be done during the workshop if at all possiblei 
or immediately after the workshop, ft is important that trainers- 
evaluators get a sense of the total picture of evaluation and do not 
confuse a part with the whole. When carefully read and understood* the 
book will teach not only evaluation of development training but also a 
tot about training design and about developnent planning itself. 

This book has emerged from a series of workshops and panels on 
evaluation of basic education and development training programs 
onjanwed by the German Foundation for International Developnent (USE) 
in Kenya during 1979-U2. 

This workshop series rad Us antecedents in a 2-week workshop on the 
evaluation 0 f "functional literacy programs' 1 organized in Mombasa* Kenya 
during Hay g-21» 1977 under the Joint sponsorship Q f DSE and the 
International Institute for Adult Literacy Methods (MALM). The focus 
of tins workshop* understandably* was on literacy evaluation; and it was 
attended only by the literacy and adult education officers nf the thun 
division of adult education of the Ministry of Housing and Social Services 
of the Government of Kenya. 
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Tt was at the 1977 Mombasa workshop that the Idea of organizing 
evaluation workshops for *»du1t educators** from all the various 
development sectors was born. The realization emerged that there were 
adult educators In other development sectors such as agriculture extension* 
health and nutrition, family planning, cooperatives and workers education; 
and that a diffusion of evaluation skills within this expanded adult 
education culture will have a salutary effect on overall development 
plans and development outcomes. At the same time, there was a sharpening 
of focus on teaching evaluation skills to trainers^ in training centers 
and Institutes engaged in PreParlno development workers and change 
agents, ft was hoped that this strategy would maximize the effects of 
the evaluation tratnine offered In Kenya. Trainers receiving training in 
evaluation techniques will Improve delivery of their own training. They 
will also he able to share some Of their evaluation skills with develop- 
ment workers they wili. In turn, train themselves. 

"Thus, emerged the series of workshops and Panels on the evaluation 
of development training programs, and with it the Action Training M.<n!el. 
The Action Training Hodel, which has been described more fully in Chapter 
TV, included the features of participative planning; development of 
evaluation proposals during the workshops and their Implementation hack 
at the training center; Provlsfon of technical assistance In short 
panel; and continuously through the establishment of an C Ideational 
Fvaluatlon Resources Committee; and emphasis on developing local ca^-iclty 
to later continue the project without technical assistance from oittUde, 
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Hi tli the evaluation workshop of June 1982, the first phase of 
tifis series of workshops and panels cones to an end* and a second phase 
begins. In a second phase of four years, 1902-86, the series of workshops 
and panels on the evaluation of basic education and development training 
wfl) be continued by the Institute of Adult Studies (IAS), University of 
Nairobi. Kikuyu, Kenya. The Action Training Model will be further 
implemented. The local organizers will receive sane funding fron 0S£ 
but no outside consultants will be used. 

Hns book, as we indicated earlier* was born in the context of the 
workshop series just described. In the first workshop during May 21 to 
June I, 1979 at Kericho. Kenya, II. S. Ebola's evaluatin g Functional 
Lite i acy (Hulton Educational Publications* Amershamt Bucks, U.K., 1979) 
was used as the text for the workshop. It became quite cleari however i 
in using this book that while transfer fron "functional literacy" to 
Me vt I op*ne nt training 1 ' was possible, it was problematic in that setting. 
In a short two-week workshop* it was best not to impose on the trainees 
the additional burden of making the conceptual transfer from literacy to 
(level upinent training and unduly complicate instructional tasks. The 
*st thing to do was to develop Materials that dealt directly with the 
world of work of development trainers, related with their institutional 
settings, and their special concerns in training and evaluation. That 

Winu. the first phase of the 0S£ program, the following workshops 
und j»,mels on the evaluation of basic education and development training 
progums were conducted; (l) Workshop during Hay 21 to June I, 1979; 
iZ) \ an el during January 4-10, I960; (3) Workshop during August 18-29, 
ly&U, (4) Panel during March 23-2&, 1981; (5) Workshop during June 1-13, 
19U1, (6) Panel during February 22-27, 19(32; and (7) Workshop during 
June ifti to July 10, \982. The first five events took place in Kericfto, 
Kenya and the last two In Naivasha, Kenya, 
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Indeed was what we undertook to do. 

We started witn handouts' dealing with the InyneJIate and expressed 
needs of workshop participants, thereby cw^lementtn^ the literacy 
evaluation text. We are Pleased that ue are now able to present thi *; 
a complete and*coff^rehens1ve treatment of the topic of evaluating development 
training programs for use by development trainers. This new book will 
find fts first use at the June 1982 workshop In Naivasha* Kenya and will 
continue to be tested In the workshops and panels that* follow.' 1 We do hope 
to use the feedback resulting from the utilization of the book for Pieparlng 
U for later Publication* 

I have had the pleasure and Privilege of directing all of the workshops 
and Panels (wl^h the exception of the Halvasha Panel of February 22-17 » 
1982) during the first phase of the workshop series. bYt these things 
are never really done singly, by oneself. In a very real sense, thp^e 
workshops and Panels have been co-directed by Or. Josef Huller of the DS£ 
who has been tty full partner both In developing the concept of the Action 
Training Model and Tn sharing Instructional responsibilities at the work- 
shops and panels. Dr. John W. Ryan* now of the Literacy Division. 
Unesco, Paris (and previous . director, 1IALH); ft", Tom MuIusa of IAS; 
and ft*, JosePh Dondo* Previously of IAS* and now of the Kenya Institute 
of Management* are the other intellectual parents of the Actfnn Training 
Hodel . 

'li, 5. Bhola, Designing and Evaluating Developm ent Iralnlno Programs; 
A Working Dojurognt (August 19801; Designing and Evaluati ng Develo pment 
Training Programs? Part 11 (March 19B1); and Dita" "Dfsp_Tays.j^r_Da_ta 
s L j^llwFT Pa n et Tf eb r ua ry 1982). 
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My sincere thanks to all of my colleagues from Kenya who have 



constituted the faculty for these workshops and served on the 
Educational Evaluation Resources Committee* especially* Joseph Oondo, 
Tom Mulusa and Daudi Nturlbi of IAS; and Alice Waka of the Kenya 
Institute of Education, % thanks also to Peter Kinyanjui, director, 
IAS who has provided insightful leadership and unfailing support to the 
project* and has now decided no go ahead with the second phase of the 
workshop series- "~ 
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such yood teachers. 
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chapter t 

what is evaluation? 



^ - Evaluation Is the process of assigning values to judge Ute amount, 
degree* condition* worth* quality or effectiveness of something. As 
human beings, we are perpetual ^valuators* We evaluate Ideas, attitudes* 
actions* characteristics and possessions of ourselves and of others* at 
home and at work* day It* and day outt sometimes sel f -consciously but 
often unthinkingly; and make judgments on the basts of those evaluations, 
IV Hfe can be oeflne as a series of decisions* then being Is evaluating. 

In recent years* evaluation has emerged as a specialization that' 
teaches us how to make evaluations that ars self-conscious* systematict 
objective and dependable* and which can* therefore* be more useful in 
making appropriate decisions In our case^ decisions about Implementing 
development training Programs. 

Let us examine a few definitions of evaluation that have appeared 
In literature over the years; 

Professor Lee J- Cronback defined evaluation simply as "the 
collection and use of Information to make decisions about an educational 
program," 

Dr, Marvin C. Alkln described evaluation as the "Process of 
ascertaining the decision areas of concernt selecting appropriate 
Information* and collecting and analyzing Information In order to report 
summary data useful to decision-makers In selecting among alternatives." 

Dr. Oanlel L. StufflebKim defined evaluation as "the process of 
delineating* obtaining, and providing useful Information for judging 
decision alternatives. 11 

l IS 
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In a recent book (See Bibliography)t Professors £gon G. Guba and 
Yvomu S. Lincoln have defined evaluation "*s the process of describing 
an evaluand [the entity being evaluated] and judging its went and worth." 

horn the various definitions included above > It becomes clear that 
evaluation involves collection and analysis of data objectivel y^ 
sy steij <jt >cd I ly and with exact! tude , to serve a variety of decision - 
making needs* These decision-Biaking needs e*ay relate to (1) modification 
or improvement of a development training program; (2) certification of & 
training program in its present form; or (3) making decisions about 
alternatives within tne program* or a total alternative to it. In sum* 
the objectives of evaluation art- always utilitarian in nature. 

I vajn a tion and Monitor inrvg 

hie word uLomtorino, has recently come into use in the literature 
of prugran evaluation. To monitor is to check uPon an ongoing program 
for flaw* or breakdowns* to enable decision-makers to regulate activities 
and to undertake corrective action. 

As in the case of evaluation, monitoring also serves the needs of 
deusion~mjkers. Monitoring* however* Is only a part of evaluation, and 
not trie uhole of it. Monitoring seeks to look at the performance data 
generated by programs In tne process of their Implementation and taut Ions 
decision-makers about the gap between rea Mty and expectation. 

hograms can be made more monitorable introducing proformasi 
*chedul«;s» and'proyress reports at important points and levels uf the 
prod^im so that the program can generate significant performance data in 
tne very process of lis implementation. Monitoring, typically* requires 



the establishment of management information systems within programs so 
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that Information generated by the program can flow to a central point 
at fixed Intervals of time. There It must be aggregated ana analyzed, 
and stored for retrieval and use at other times, 

Monitoring and Supervisio n 

Supervision Involves some monitoring btit* at Its best* supervision 
Is an educational Process. Good supervision enables the various function- 
aries tn a program. to analyze and evaluate their own performance In relation 
to the program needs* and t6 learn and,grow on the job. 

E valuation and Researc h 

Evaluation and research are two different Professional activities* 
though the two get often confused. Confusion occurs because the 
evaluator and the researcher share Inquiry deslnnsi methodologies, tools 
and Instruments* and have similar concerns with the validity and tht* 
reliability of their findings* Quite often the same one person may be 
acting as evaluator and researcher at tne same timo. Evaluation an«1 ' 
research* however* differ significantly In terms of their frameworks 
and their objectives, as the following table should Indicate: 

Table 1, The different orientations and objectives of the evaluator 
and the researcher 

t va 1 ua tor forcea rcher 



Policy and planning orientation; 
seeks to clarify planning 
alternatives and to Improve 
program Performance. 



OlscIPl Inaryand academic 
orientation; seeks to advance 
the frontiers of knowledge In the 
researcher's chosen discipline. 
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Loyalty is to a particular 
program or project; choice of 
evaluation topics is determined 
by tltt information needs of 
dec is ion -makers. 

The methodo logical choices are 
often quasi-experiiuental or 
non-experimental i the norm for 
judging the findings is 
applicability to the progran 
situation and adaptability to 
other similar program settings. 

Tine-frame for the production of 
results is set by the program. 



Professional rewards, consist in 
ihe utilization of findings by 
decision-makers and demonstrated 
improvement in the program 
implementation. 



Loyalty is to a particular 
academic discipline; choice of 
research topics is determined 
by the theory and research 
needs of the discipline. 

The methodo logical choices are 
typically experimental or 
controlled; the norm for judging 
£he findings is generalizability 



lime-frame for the production 
of results is set by the 
researcher and by the internal 
logic of the research question. 

Professional rewards consist in 
publication of findings* in 
professional journals and 
favorable coimients by 
professional colleagues. 



Ihe distinction between evaluation and research is not, however, 
water-tight. Ihere is a lot of had research around. And there are 
sane really good evaluation studies that produce insightful and 
generalizable findings and get published in prestigious professional 
Journals. Such evaluation studies may be seen to fit the title of* 
"evaluation research/ 1 the phrase used by Carol Weiss. 



I nternaj^ aiid^ External Evaluation 

< Internal evaluation is that conducted within the program system 
by program specialists themselves. External evaluation is that conducted 
by evaluators sent frcm outside. 

It is often asserted that external evaluation is more objective 
thcin internal evaluation which is rejected as both subjective and 
political. On the ' oUwr handt program specialists often dread external 
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evaluations which they complain are often hurried* superficial, 
uninformed and political. There Is merit in both Positions 

Exti nal evaluations are by no paeans Inherently objective. Oft the 
other hand, Internal evaluations can be most objective. Dotii intem.il and 
external evaluations can be highly political. The right approadi U to 
assign appropriate tasks to Internal and external evaluations, and to make 
.them to perform complementary roles. 

In this book, our emphasis Is on internal evaluation which we look 
at as evaluation for growth rather than for control. Gy conducting 
internal evaluation. Indeed an evaluation of one's own training performance 
by oneself, the trainer can perform the essential most task of oil 
evaluation which is to Improve performance; and. In the Process* can 
grow as a professional trainer. 

Participa tive Evaluation * 

D articiPative evaluation goes beyond the formalism of both internal 
and external evaluations and conducts evaluation in Participation with 
the publics concerned. Evaluation becomes both educational and 
liberating. Essentially, participative (or participatory) evaluation is 
one that is conducted in mutual collaboration by all those engaged in 
the conduct of a program. At its best, the organizers play a 
facllltative role while the people being served by particular Proqraftts 
take over* These people determine* through dialogic action, what the 
evaluation needs are, what Information should be collected and how, and 
what norms and standards should be used to Judge success or failure. 
Participative evaluation Is a methodological approach with otrowj 
Ideological r.onmiitment to radical humanism. We will haw more to s.iy 
"about it in a later chaPter, 
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F onna ti v^ and SuimiatW e Cvajuatjons f 

V*± concepts of formative evaluation and suiunaUve evaluation, 
introduced by Professor Michael Scriven, are important for evahtators 
to understand. 

formative evaluation examines different aspects of a program as (t 
is being formed. It is used to correct and Improve a program during its 
planning and Implementation, \ r 

Suimiative evaluation typically comes at the end of a program or at 

i 

the completion of a program phase. It is used to sum up the impact of a 
program on Its client groups or con muni ties. 

Product Evaluation 

Product evaluation may be seen as a special case of formative 
evaluation when what is being evaluated is an instructional product* such 
as, a textbook* a set of posters or charts* a film or a simulation-gane, 

Sujiinar y 

Evaluation is the process of assigning values tn judge the amount* 
degree* condition, worth, quality or effectiveness of something. As a 
specialization* evaluation involves, the collection and analysis of data 
objectively* systematically and with exactitude, to serve a variety of 
decision-making needs, (valuation can be usefully distinguished from 
monitoring, supervision and research. Evaluation may be Internal or 
external, or way be participatory in nature. Evaluation may be formative f 
(conducted during the formation of a program) or suwaative (conducted at 
the eta! of a program to sum up results). 
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Thing s tj^ Po or Jhlnk A bo u t 

1. Are you clear about the distinction between evaluation and research? 
Try to M*1ain the difference between the two to a colleague to Ms or 
her satisfaction, 

2. From your area of work, think of a topic that seems to fit the 
category of formative evaluation. Think of another topic that fits (he 
category of sunroative evaluation? 

3. What kinds of information must be generated and collected for 
developing a ironftoring system for your training center or Program? 



CHAPTtH II 

A GEHLRAL MODEL OF DEVELOP KENT 



If we are interested In the evaluation of development training* we 
have yot to understand all of the three processes Involved — evaluation* 
training* and development. Therefore* In presenting a general model of 
the process of development* we are by no means straying away from the main 
concern of this book. 

Inert are two good reasons why a person preparing to be an evaluator 
of development training programs should also understand development. One 
is a general and somewhat abstract reason* but the other is downright 
practical and functional. Tralners-evaluators engaged In the preparation 
of development workers and change agents must understand the processes of 
♦ development to #et a sense of the framework within which they work. They 
^ must observe the big picture and must understand the Interconnections 
between and among the various development actors and development 
processes. It is only with such an understanding that their concrete 
training actions* their training methods and the training effects they 
obtain* wiM make sense In the larger perspective of change at the societal 
level. And, it Is such an understanding that will help than to be more 
than mere technicians or purveyors of skills and Information but will 
make then enlightened and committed change agents — indeed culture-makers. 

A SfeLond reason why tramers-evaluators should understand development 
is more practical and functional. Since training seeks to prepare workers 
to do better development* it Is developnent that will provide most of the 
norms and criteria for Judging the success or failure of training designs 
and training method*. 
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A Genera 1 Hode_1_ of Deve jofwnent 

A general model of the development process H Presented below: 



Culture at / 
Time. \ 



Institution Buildlitfjf 
(Economic, Social and Political ^ 
Institutions) 



Ideology x Technology 



Hunan Resource Development . 
(Through Formal* ttonformal and f 
Informal Education) 



Figure K A general model of the process of development 



Culture at 
Time, 



The process of development and planned change (as a culture goes 
through some sort of transformation during a particular time period of 
history), Is essentially a calculus of Ideology and technology! an 
adaptation of means to ends* This essential process of Interactions 
between Ideology (ends) and technology Wans) can be seen to factor out 
along two itimonslons: (1) institution building and (2) hunan resource 
development* 

While development Ideologies may differ somewhat from country* to 
country, a basic core of development is coming to be universally accepted. 
Development has to be man-centered* Development should Involve not merely 
greater production, but also a J»*t distribution of what is produced. 
Social and political Institutions should he responsive to the needs of 
the people and should provide opportunities to the People to Participate 
within tliose instituting 
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Technology is what enables societies to implement their ideological 
visions and their societal aspirations. Technology itself is of two major 
categories: physical technology which transforms raw materials into 
econonic goods; and social technology which trains and motivates people 
and invents effective social arrangements. 

T wo 0 linens ions of the Development Process 

As we, have suggested above the "ideology x Technology* 1 calculus can 
be seen to occur along two dimensions: institution building and human 
resource development. Institution building is the name given to the 
process of improving existing institutions and designing new institutions 
to undertake the societal tasks tnat need to be done. Whatever needs to 
be done systematically and with some continuity needs a system — an 
institution of some sort. No wonder* institutions have been called the 
building blocks of cultures. 

The new institutions required by development societies are many. They 
are economic, social and political; parliaments, primary courts; 
cooperatives, radio forums, classrooms, rural credit societies, uianu- 
facturing plants and factories. 

But new institutions wil! be no more than aapty shells if they do not 
have trained people toman those institutions. Tnat brings into fouis the 
role of human resource development. Htiuan resource development may bo 
undertaken througii three different but complementary educational strategies 
formal education, nonformal education and informal education* 
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chronoTogfca11y-graded modern 'educational system* that stretches from 



to socialization: neither the teacher nor the learner are self-conscious 
about the process of teaching- learner taking place. Finally. nonfcimil. 
education refers "to the motley assortment of organized and semi-organized 
educational activities operating outside the regular structure and routines 
of the formal system* aimed at serving a great variety of learning needs 
of different subgroups In the population* both young and old." 1 it Is, 
thus* In nonformal education that the trainers of development workers 
should be most Interested. Trainers of development workers are Indeed 
nonformal educators. 

Oth er. Th a n Train in g Solutions 

The modeT presented In this short chapter should have helped us put 
training in a proper development perspective. Training has been shown to 
be a significant component of development. At the same time* the general 
model of development presented here should have helped us understand that 
training Is not the solution to all development probTems. Problems may Tie 
In defective national visions* * n lack of resources of raw materials* In 
tho Inadequacy of physical Infrastructures* or In the very design of social* 
economic and political Institutions. In these cases solutions win He 
elsewhere than In training. J 

'The definitions of formaT education and nonjformaT education given 
here are taken frtm Kanzoor Ahmed and Philip Coombs (Eds.). Educa tion^ 
for RuraHtevelppntt n J:^ Ca se Studies for Planner ^. Hew York* fVaeger 
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Summar y 

Trainers-evaluators should be Interested In development to understand 
the true nature of their role as development workers as well as to 
develop the criteria to judge the success or failure of their training 
designs. Development basically Is a calculus of Ideology x technology or 
the adaptation of means to ends. Mother way of defining the development 
process toll? he in terms of (1) institution building and (2) Um&n resource 
development. Human resource development* especially through nonforntal 
education* is of utmost importance In the overall development process, 
Trainers-evaluators* however* need to be aware that all development 
problems are not solved by training* or by more and better training. 

Tilin gs to go p£ Think About 

1. What are the national aspirations for development In your country? 
Separate thou as ideological (related to ends) and technological (related 
to means). 

2, Uhat kinds of development training needs are being fulfilled in your 
country by the formal education system? 
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CHAPTER 111 

lHt PAOHISE OF EVALUATION IN 
DEVELOPMENT TRAINING PROGRAMS 

lhe promise of evaluation In development training Programs Is rooted 



In the scope and significance of training In the overall development 
processes. 

The_ Sfi e and Scope of 06 v cl0pment_Trajnlnq 



He need to start with the training (or educational) needs of our 
people, lhe educational needs of people In the developing world are many 
and varied. They are political, social and economic; and they are 
Informational, attltudlnal and skllls-relatedr Farmers* workers and 
housewives In the developing nations must learn new developmental values, 
attitudes and aspirations. Fanners must learn new scientific agriculture 
and animal husbandry; and new modes of farm management, cooperation and 
marketing. Workers must learn new technological skills* nKwe Productive 
work habits and patterns, and safety rules at Places of work. Housewives 
must learn new ideas about famHy care and nutrition. And they should 
all — farmers* workers and housewives « learn to participate In the 
processes of decision making that govern their lives and work. 

8ut how to make all this possible? Clearly, people can not be left 
to themselves to 'earn all this on their own. At the same time, we can 
not bring them all to school. In fact, we do not have those many schools 
to accommodate them. Since we can not brfng all those people to school, 
we must take the school to them. And that Indeed Is what is being done fn 
all kinds of places, in all kinds of settings* all over the Ihird World 
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through what we have called nonfoimal education, 

Jo provide this Needed education to these new clients and consumers 
of education, we need new educators . We have called than development 
workers or change agents or front-tine workers. There are other names 
fur them as well: coiimumty development workers* agricultural extension 
workers, vtltayt health assistants* barefoot doctors, cooperative eduutiun 
workers, family planning workers, nutritionists, political educators* 
primary court jud*jest workers' educators* literacy workers, radio and film 
forum leaders, discussion te^<"s and animators. All over the Third World, 
these new ro jes are being created to educate the people* to develop the 
huuan resources of nations for the ultimate transformation of societies. 

The conceptualization anu design of these brand new development roles 
has been one of the most challenging tasks of development work. What 
should those roles he: agricultural extension worker, cooperatives , 
assistant family health educator,, family planning specialist, political 
educatort or village level multipurpose worker? How should change agenU 
he ructuited to these new roles? What should be their orientation to 
social change? Should some promote diffusion of knowledge and skills* and 
some others promote critical consciousness? How should tasks he divided 
atiiong and between tlitta? Jf tasks are divided on the basis of specializa- 



tion, how snuutd integrdtion be achieved in the delivery of services? 

The training of these new educators is thenost important question of 
alt. How to train change agents for effective perfottnance of their role*? 



socialise these change agents In the n&w norms and values that are necessary 
for them to internalize to serve the disadvantaged? 
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The above by no means exhausts all the training needs or development. 
The trainers of front- fine workers must themselves be trained to perform 
their training tasks, and so on up the Hne. The following simple diagram 
should help us visualize the extent and scope of the training problem: 



Trainers 
(12} 

of 

Irafuers 
(TO 



Trainers 
(11) of 
frontline 
development 
workers 



Development 
workers at 
the 

front line 



Learners 
In 

rural and 
urban 

communl ties 



Figure 1. The trainer- trainee chain in the human resource development 
approach to development. 



The formal education system has been unable to handle the training 
tasks necessary to implement the total human resource development app.rjtia^.h . 
Only Type T2 trainers have typically come from forWJ Institutions of 
higher learning — sociologists, anthropologists* educators* media 
'PeciaHsts, etc. — who* at their best* had acquired experience in 
development by working In the field on various development projects: and* 
at their worst, never had any touch with the development world and were 
discipline-oriented in their training designs. 

Type Tl trainers have been trained* as could be exPected, outside 
the formal education system In specialized institutions. Such institutions 
have proliferated all over the Third World and are now consuming quite an 
imPresslve proportion of national education and developmental resources. 
The D1rectorv_of _fe1u1t _Edu,ca t1o_n Jjnsti tut ions In Kenya (A Guide on Schools, 
Centers, Colleges* Courses and all Institutions Catering for Continuing 
Education In Kenya), Issued by the Board of Adult Education, Nairobi, 1976, 
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lists a variety of training institutions that include: institutions of 
public administration* business management* and development adwi nitra- 



tion; of agricultural extension training animal health and husbandry; 
institutes and centers for daily management, meat packing, tea, and 
forestry; cooperative colleges and centers for water development; 
institutions for the training of workers in health and family planning; 
for tiome economics anil nutritions resettlement of cripples; institutions 
for training personnel for literacy centers, community centers, village 
polytechnics, vocational centers, far&ers training centers i <ifld centers 

for training personnel for evening schools, correspondence colleges, 

*■ 

extra-mural programs and mass couwunication. 

TJie size of the training sector in development is today an important 
indicator of the seriousness of the development effort in a country and 
of the possibility of its success. The effectiveness uf the development 
training sector and its efficiency are thus important considerations. * 

The Promise of Evaluation 

The promise of evaluation in development training can now be well 
understood, Effective training of various trainers in the long trainer- 
trainee chain, and especially of the change agent at the front-line, is 
crucial. The size and scope of development work is such that good 
tiaimng can* make an important difference tu what ultimately Jiaoocns to 
our schemes regarding food production, health and nutrition, family 
planning and political participation. 

but to Know where improvements in training are possible we need to 
undertake systematic evaluations of development training, here is a set 
of questions that are crying for answers: 
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1- Had the new change agent roles designed to promtfte development within 
communities proved effective? 

2. Had the Institutions and programs established to handle training of 
new change agents performed their tasks effectively and efficiently? 

3. Had the procedures for recruitment of change agents for development 
rotes* the training curricula designed for their training, and Instruc- 
tlonal methodologies used In their training been correct?, antf, finally, 

4. Had the overall training effort made any significant Impact on the 
11/es of people whom the trained change agents were supposed to serve? 

Answers to these questions will bring high returns on the Investment 
0 1n development training. Herein lies the premise of evaluation, 

Swnjary. 

The development training enterprise ts significant both In size arid 
scope In most development countries. Improvement In the effectiveness of 
training can generate high dividends. To know how effective current 
training programs are, they should be evaluated. The promise or evalua- 
tion In development training Is Immense, 

Thin gs to Do or Think About 

1, On the basis of your personal experience In development* wlial groups 
- of development workers In your country you think Is performing at less 

than a satisfactory level? Can more effective training help In thai case? 

2. In your opinion, which Is the weakest Hnk In jthe trainer- trainee 
chain within the development training sector In your country? 



CHAPTtR IV ; 
DESIGKIHG THAI KING SYS f Oft FOR 
ttfVtLOPMfcNT WORKERS AHP CHANGE AGEHTS 

: Td evaluate a thing, an Instrumentality* a process or a system, we 
mtist first understand that tiling i instrumentality* process or system. ( 
The first task that we face as evaluators of development training, is to 
obtain an understand ing of what development training is. 

After understanding comes description . We must be able to describe 
the development training' process. Since our ultimate objective In 
evaluating development training is to Intervene in the training process 
to improve it, our description of the development training process must 
, be made ift desi gn terms. 

E dyca tj Of| aji d Mrajn jnti 

Distinctions have sometimes been made between education and training. 
Uhile such distinctions may sometimes be quite useful, they should not be 
considered absolute. Both education and training* after all* are concerned 
with human learning and 'performance. p 0 th seek to teach new values, nek 
attitudes, new knowledge and skills, m this respect, education and 
training are similar. Itowever, the two are dissimilar In one important 
way. Education is preparation for life In a society . Therefore, 
objectives of education are general. Training, on the other hand, is 
preparation fur work in a particular structure or s ystem . Therefore* 
objectives of training are much wore particular. These distinctions are* 
by no means* sacred. Training, at Its be*t» becomes education. 

J? 
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. Training Is* typically, defined In reference to some organisational 
objectives: a plannejd process of teaching and learning that seeks to 
prepare rottribers of a group or an organization to acquire knowledge, 
attitudes and skills required by the group or the organization for imple-. 
menting Us objectives and, thereby. Its overall mission. 

Tr a .jjjjjg Jte igl:^ .Theory and Tradition 

Training design theory can be seen to be rooted fn curriculum * 
development theory. More recently training design has become directly 
linked with Instructional development theory. Iralnlng design as practice 
has liad a long tradi tion 'In the U.S. Army and In American business and 
Industry. That Is where most of Its operational models and methodologies 
seem to have been developed. 

The following six principles have emerged from research and Practice 
on training within the U.S. Army over the years during and since Morld 
War ll» as summarized by ttushnell;' 

1. Performance -Ba sed Instructi on^ Students should learn the skills 
necessary 76r lob performance. The emphasis should be on active stilt 
practice -- "doing* 1 rather than "passive" adsorption of Information, 

2. Task M astery . Every student should be required to reach a particular 
standard of fie r forma nee In each still. Assessment should be on a H gn/ 
no-go' 1 basis. Tlie student who does not reach the Criterion level of 

^performance should receive additional Practice until he does reach it* 

3, Functional Context. The student learns best In a job-relevant 
situation" TEeorcti cal or technical material should be presented only 
when it Is needed In learning to perform a skill, 

4, Individualization. For various reasons people team at different 
rates, "To the extent"|joss1b1e, the trainee should he permitted to learn 
a skill at his own rate* 
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Davis 5, Oust me tit "iralnlng as a Knowledge Production and 
Utilization Strategy: InstructlonaVTethnology In the U.S. Army/' 
Viewpoints in Teachfng'and Learning, 54(2) : 114-127, April, 1970, 
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$. f eedback . If the instructor and curriculum developer know a good 
deal about the student to be taught* instructional methods can be modified 
to be more effective. If the student knows about his own skill acquisi- 
tion, lie will be motivated to correct errors of Inadequate performance. 

6- Qual ity Control . To ascertain that the training system is function- 
ing properly* student performance muU be systematically assessed at 
various times during and at the end of training. 

Training within the army has been heavy on technology — films* 
television* video* and computerized simulations. 

In the business sector and in the few universities that ai'e getting 
interested in the training technology* the new vocabulary is of front-end 
analysis* performance discrepancy* learner analysis* cognitive styles* 
task analysis* criterion-referenced testing* mastery learning* job 
engineering* environment analysis* contingency management * feedback systems 
and organization development. 



Results obtained from the application of training design models and 
techniques listed above have been impressive within the formal organiza- 
tional settings of the army and industry. Indeed, concepts such as 
front-end analysis* learner analysis* task analysis, individualization* 
feedback and criterion-referenced testing could *U be usefully employed 
in the sector of development training — training of development workers 
<tnd change agents, however, one will have to be extremely cautious about 
a wholesale transfer of training models and techniques from the army find 
industry to sot to- economic and political development within Third World 
communities. In the development training sector* the environment of work 
is fluid* and work roles have to undergo a process of adaptation Which 
makes job engineering quite difficult. Learning by doing as a training 
strategy is not always possible within community development Settings and 
role performers have to grtw on the job. In this sector which has always 
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be*m subjected to severe scarcity of resources, use of audio-visual media 
and simulation technologies has also been rare. <- 

■ 

!?_ e _ .5p_?ct al Chara cteristic s of Deve lopme nt _ Train fn g 

* There are some special characteristics of training in the development 
work fn general over the Third World. Some of these are discussed be'ou: 
1* Unlike the army or Industrial training* training for development 
work has to deal with systems that are open an d somewhat undefined. 
Development workers have to be taught to be sensitive to social systems; 
they have to learn to be c ultura l ly-or len i ted . 
2. Development training also has to Involve a component of needs 
assessment and n eeds negotiatio n processes. Development workers can not 
tafce^e-Pacfcaged solutions to pre-determlned Problems. In each new 
rowminfty* the development worker must go through the process of 
determining the "felt needs" of people and then "fashion" new profiles of 
needs. PartTclpatlvely with the local conmunftles* through a Process of 
needs negotiation* Involving a dialectic between needs as seen by lucal 
communities ( ind needs as defined by the developmental elite. 
3- It follows from the above that training for development has to Include 
both substantive and process compii^nts. Substantive knowledge In 
agriculture, ftmfly planning, health or Intermediate technology fs not 
ennugh, the development worker must nave been trained also In communication 
and social Participation skills. 

4. Development training often Involves the three-told Process Of role 
invention* role f Iffij tat ton and roj e soej aJXgatlon ■. Development workers 
oitpn have to face the challenge of Inventing new roles and Introducing 
tJK<K hi conmimttles as they assist In capacitating ard socializing 
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workers for those roles. 

f>. Typically- training is defined as preparation for role performance 



working in isolation within one organization or structure will be a 
disability. Development training has to involve an understanding of 
system Interfaces and an ability for coordination . 

6. Development work ts In a very real sense political work. Invariably 
development involves new distributions of power, status and economic 
goods to Include the marginal and the underprlvlledged. To promote 
the welfare of the oppressed and the disadvantaged ► especially within 
economies of scarcities- requires strong ideological commitment on the 
part of workers themselves* thus development training must Involve a 
component of value c larification and indoctrination . 

7. While training always lends souie legitimacy and status to the 
trainee. In the area of development training the allocation of status 
tan have significant structural consequences within the community. The 
recruitment patterns used In the choice of trainees may perpetuate exist- 
ing power structures or may introduce a new generation of non^ jrad j tionaj 
leadershi p from new social classes. 

fj. Uhfle effects of and returns from training within Industrial and 
business settings could perhaps be measured in terms of dollars or 
shillings or some other currency* return s from development traini ng are 
delayed and are often ambinunus. Development trainers have to develop 
some Intermediate criteria of success for their training, 
9, final Iy* the realities of the World of development and of the Third 
World rural areas In particular* make control * supervisi on, follow up ami 



within a structure or an institution. However, In the development world* 
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feedback quite difficulty If not Impossible. Recruitment of trafnc-es 
Itself can not always be controlled* they may b* self-selected. Nor Is it 
always possible to control trainee performance. It Is not possible to 
Integrate voluntary development workers within organizations and to super- 
vise their work. Lack of Infrastructures of communication often make 
follow tip and feedback well-nigh Impossible. 

A H gd el^ o J _jtev elopment Training Oesign ■ 

Training design for development settings requires "system thinking" 
on our part. What Is a system? And wn?t Is "system thinking'*? 

A system Is an orderly arrangement or combination Of Interrelated 
and Interdependent Parts or elements emerging Into a whole, A family is a 
system, A cooperative is a social system. A training Institute Is a 
social system — maybe we should say a techno-soclal system. 

System thinking Is the mental habit of looking at things, whole. It 
Is "holistic thinking. 11 it Is the type of thinking that enables us to 
avoid getting caught Into one-to-one relationships* and linear extensions, 
it Is thinking that involves "atonceness" and learning to see systems L 
where others may see mere individual entitles. 

System theorists have studied the Properties and characteristics of 
systems, ihey have found that all systems can be described In jterms of 
four parameters (or guidelines): inputs t processes* contexts* and otitPuts. 
That 1st ail systems receive some InPuts which are subjected to some 
processes* under particular contexts t to produce some outputs. The ability 
to use the same four parameters to describe all systems is most convenient 
ami a most useful consideration. 
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A generic model of training design in developmental settings is 
presented on the foflotumj page. It uses the system metaphor and the four 
system parameters inputSi processesi contexts and outputs. This model 
should help trainers in the developmental sector to organize all the 
variables that can or do enter a training situation. They can then look 
at those variables that offer real ogt fo ns for manipulation within a 
particular training setting. If options are available* they can thfn 
evaluate variables under different conditions and put evaluation to work 
to actually improve the performance of training systems. 

The model presented here shows relationships between and among three 
systuns the training system^ the performance system, and the social 
system. The outcomes of the training system, with additional inputs XH 
{laudroversi medicines, high-yielding variety seedsi etc.) become inputs 
for the performance system. In turn, the outputs of the performance 
system with additional inputs PQj* (ideologies* trained planners* 
infrastructures, etc.) become inputs in the larger societal system engaged 
in development. 

A systemic view of development training* with mutoal interfaces 
between the training system and the program system* and then between the 
program systau and the social system, should enable training desfgners to 
ask Mue^ttons about the development roles that should be desifjncd, and 
about the technical and social content of those roles. 

S teos \n_ Tr aining Des ign 

Jn the following* we have translated the general model of development 
training design intu "action terms" ond have suggested six steps involved 
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figure )* A model of development training design and evaluation* (6hofa, 1980) 
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in training design: 



STEPS in TRAINING DESIGN 1 



Step 1 * Social and program analysis for the definition of both 
general and specific development objectives 



Step fl Listing program activities to be conducted for the achieve* 

uent of defined development objectives 

Step HI Defining and designing roles and allocating role responsi- 
bilities for conducting listed program activities 



Step IV Statement In behavioral terms of competencies required of 
various role incumbents; and listing of new attitudes and 
values needed for socialization in those roles 
* ■ * 

Step v Developing a training curriculum to build required 

competencies and the implementation of this curriculum in 
training courses* works'nops* cm- the- job training and 
correspondence training 



Step VI Evaluation of the effectiveness of training and review ol 

steps I to V. 



One should noce that the steps for^tralnlng design listed above 
encuiipass all the three processes of role deslgnt training for the roie 
and socialization for the role. 

by way of a siWuary, the concept oir development training design can 
be defined as follows: 



1 

Adapted from H* flhola, "Description and Evaluation of a Training 
Progr^idift* for Literacy Teachers and Their Trainers," Indi an Jour nal of 
flduj t Educat ion. Vol. 31, No, 5 ( Hay 1970, pp. 3-6, iCTbv 
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- 4 To train is to Instruct and socialize 

- in relation to a defined set of organizational 
objectives, 

* using appropriate c^jjent and methods, 

■ in appropriate settings; 

■ within a te^orary system 0 r an established 

Institutional structure, 

* an appropriately selected group of functionaries, 

generally from a larger pool of such functionaries, 
using some particular criteria; 

* for effective role performance* 

* at some anticipated level, 

* within a matrix of organizational and social 

realities of the 'world of work; and 

* with expectations of feedback for trainers and of 

self-development by trainees on the job, 

"Havelock and liavelock' list four steps In training. design as follows* 
L Planning 

2, Defining objectives 

3, Specifying learning that should meet objectives, and 

4, Specifying the sequence of training activities that should lead 
to desired learning. 

They then go on to list eight principles of good training design; 
1. Relevance to personal background and back-home situation of trainees 
2 t SpeciJJcUy of objectives 

3, Generality so as trainees will be able to cover a broad range of work 

Donald Have lock and Marcy C. Havelock, TraJn^^f'brJJhange Agents, 
AJ>u1de_io t1fe_Des|gn of Training Progra ms In Educational^ Other_FleTds\ 
Ann ArEor, Hlchfganf Center for Research on TjEHlzaUon of Scientific 
Kiinwlatqe, University of Michigan, Jg73, 
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situations an J problems 

4- Re 11! to cement and retards for appropriate responses ant! actions of 
tf a fnees 



6. Cost-effectiveness of training 

Syftfar^ (use of a variety of media and methods)., and 
tt. I raisf erab lhtv of learning and skills from the training situation 
to the work situations* 

Another training specialist* Odiorne^ focuses on the training process 
and provides suggestions for "Instructional development" in training 
settings: 

1. Specify the desired terminal behavior or the exit requirements of the 
trainee. 

2. Specify the present level of behavior and performance of the trainee. 

3. using (f) and (2) above, define clearly the specific behavioral 
change required in the trainee. 

4. Provide situations within which the trainee can engage in active 
action such as talking* conferring* writing* operating* demonstrating. 
6. In implementing {4) above simulate behavior sought on the job as 
closely as possible through role plays* case studies* management games, 
and demonstrations. 

0* Guide the course of change of behavior through an order'y progression 
of small steps. 

^Georges. Odforne* Iraininsi by Ob jectives , New York: The McMillan 
Co., 1570. 
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7. Provide feedback through knowledge of results at each Possible stage 
of training. 

8. As the course instructor* stay In control of the learning procedure. 

9. Measure actual outcomes through evaluation. 

As can be seent the three sets of Instructions listed above conple- 
ment each other. Bhola's suggestions and steps rafse the design questions 
tffthln the larger framework of development Programs and training 
Institutions., The process of "role design 1 * is considered by Bhola to be a 
significant part of the training design process In development settings. 
Havelock and JJavelocfc focus on the training system, paying due attention 
to the world of Performance where the trainees will go to work. Odiorne 
discussed training design as an Instructional development problem. Each 
of these three aspects must, of course, get due attention In an overall 
training design effort. 

It Is the task of the training designer to bring together* In 
different valuest the many variables of a training system Into an Ideal 
mix to create the best training plan under a given set of circumstances, 
fhe training Plan should be such that: 

1. It Is Ideologically and strategically congruent with the national 
vision and the national development strategy. 

2. it uses existing national Institutional resources for tralnfng 
where such resources exist, and establishes new institutional mechanisms 
where such mechanisms are needed. 

3. it chooses training objectives, methodologies and durations for 

t training that the society can afford fn terms of resources that will have 
to be irccil. 

45 
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4. , it relates directly with the cultural and socio-political realities of 
the setting in which trainees will perform after training, 
b, it is based on an understanding of the learners and comnunlties to be 
served in regard tu their values* motivations and their' own perceptions of 
needs and priorities among such needs. 

t>. it takes tii view the existence of other change agents and institutions 
concurrently working in tlie field and with whom coalitions and collabora- 
tions may nave to be buflt, 

7- H seeks to teach trainees both substantive knowledge and process 
skills using appropriate materials and methodologies in well-chosen 
training settings; and 

b. it maximizes outputs of the training systou In terns of trained 
development workers* experienced trainers* and tested training strategies 
and training research* and minimises the total array of training costs, 
t^ee the "Training System Analyzer" Included as Table 6,2 in the monograph/ 
Cur iciiKrii j^vc ^o^itent for funct i onal literacy andjtonformal Education by 
ill. S. Ghola> 1979* pages 141-148. ) 

Suite Traln im] Models la Use 

Jn the practice of training design In development settings* sane 
special models for the delivery of training to development workers and 
change agents have emerged over the years. Me will here refer briefly to 
three of the>e models: 

1. The Operational Framing Model 

2. The Participatory Training Hodeh'and 

3. The Action training Model. 
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Ope ration al Uah iing modei 
The operational seminar 1 aPProach was developed within Unesco over a 
period of many years In the context of literacy Promotion for development. 
In the words of de Clercfe: 

**T1ie operational seminar may be described as a training exercise 
which enables the participants to experience, on a reduced tfnie~scaTe but 
in a real situation, the sequence and correlation of operation, which taken 
together* constitute the Process of literacy work llnfced to development, 
whether the development be predominantly economic * social or cultural, 
tike research workers operating In problem situations* the participants 
rediscover for themselves, through an Individual and grouP effort of 
research and analysis* an educational strategy geared to a development 
strategy and the fundamental concepts of a new type of literacy work 
They acquire direct experience of ways of preparing and shapings 'mode- 
to-measure' programme rooted In the requirements of a specific mil ten, 
and a Practical knowledge of experimenting with educational activities and 
assessing the various factors In the learning process. ... Theories and 
concepts are constantly held up for comparison with the Practical realities / 
df a particular environment." 

The operational Senear methodology thus seeks to be the methodutogy of 
"method demonstration" or "demonstration by result." The participants* 
through their involvement In a real-life task* are supposed to hecowe 
Instrumental In thMr own training 

Since 1970* more than SO operational seminars have bren organized 
over the world* covering some 3*000 trainees. Participants have fnchided 
all levels of cadres* ffjem planners to front-line workers, agricultural 
technicians* health technicians* adult educators and social scientists 
all in the same one seminar group. The seminars typically last 3 weeks 
though seme have been shorteri and many have been longer* Participants 
work In mult! disciplinary teams within a Particular community or zane of 

activity. The level of direction of their activities depends upon their 

*h 

'llarcel de Clerck* ihe Operati onal Soulnar: A Pioneering Method of 
Training for Ucvejopment. Parts: "Tlnesco* 1975* 
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needs. A hna) report of the work of muKidisciplinary teams is produced 
dt Ure end of the idiiinar. 

While Uie model is well conceptualized and has undergone a long 
period of testing, it has suffered from some serious problems in terms 
of professional guidance of participants; in regard to entry and exit from 
communities; relationship between theory and practice and, therefore, in 4 
reyartJ to transfer of learning frau the training situation to the work 
situation* und about the Quality of learning as demonstrated by the reports 
produced by groups. 

2.^ I he partici pator y training mode | 

Paolo freire* the Brazilian philosopher, educator and author of 
i*Si!i!tl£JUZ- g£ the Oppr essed (herder and Herder ► New York ► 1972), is the man 
who scratched the word "participation** indelibly on the conscience of 
developnent planners, trainers and evaluators. AH the world over* groups 
of ppojitti tuve worked with Paulo Freire's ideas and extended and enriched 
them through iiwplementatton, 

i 

Kamla Miasm's work in the area of participatory training also builds 

upon freire's ideas and nas the merit of being wel I -documented. 1 

me methodology of participatory training is non-directive > dialogical, 

and experiential. The fol taring » in her own words, are the main features 

t 

of her approach: , * 

Vumla Bhasin, Participatory Training for fleye jownen t . Bangkok* 
lhailandi f AO* 1976". ~K\ si o EyHcarol a flKasTn ,~jj- reak i njjB a r r i er sj^. ft. Sou tit 
Asian txucrience of Training for Parti cipatory~ Deve lopment. Bangkok* 
TTiaTfand; fWTfl?f. — — 
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1. It was attempted to Involve the Participants In Identifying their 
own training needs, In planning and In running the programme because the 
best way to teach about 'bottom-up planning*, 'people's participation', and 
'decentralisation of authority* Is by Practicing these Ideas. 

2. The emphasis was on self- training and group learning through 
continuous Interaction between participants and on exchange of experiences 
through group-discussion and actual field visits. This was done on the 
assumption that by exposing the participants to the realities of rural 
development In their respective projects, they would be able to 

- Identify the Issues and Problems that confront them as change 

agents; ' 

- arrive at solutions which are most applicable to their own 

situations; and 

- act these solutions within the framework of their own 

organizations and the communities in wfifch they work. 

3. All learning during the programme was related to the concrete 
Problems of rural areas and their possible solutions. The raw material of 
learning was provided by the participants' actual experiences in the course 
of their work. This was achieved through case study presentations, field 
visits and analytical discussions. 

4. The emphasis was on grouMlscusslon rather than on lectures. 

5. Attempts were made to create an atmosphere In which the partici- 
pants would feel free to express themselves and be ready for frank reflec- 
tions and analysis. In other words be prepared for an honest* even If 
painful Process of self-searching, criticism ^nd self-criticism. 

6. it was attempted to run the Programme Itself *"to a process of 
living together in a collective spirit. 

7> An ongoing evaluation was built Into the progranine. 

The main components of the training methodology were the following: 

- Participation ot trainees In Planning and running the training 

- Collective ifvlng 

- Case study writing and presentation 

- Field visits 

- Group discussions* 



3. The action tralning^jijodej. 

The action training model (ATM) merged within the context of a scries 
of training workshops and panels on the evaluation of development training 
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program* conducted under the aegis of the German Foundation for 
Jntuiiational Deve)o|*ient In Kenya, dur* >g 197y-19t*2, 

The following are the important features of the ATM: 
!• A Iowa period pf pre-planning and Preparation , An important feature 
ol the ATM is the long period of pre-planning and thorough preparation for 
the planned workshop on seminar. The organizers prepare themselves not tor 
one particular pre-packaued workshop* but for an unanticipated workshop! 
any one workshup out of a series of possible workshops. Such preparation 
makes it possible for the workshops to be parttcipatlvely planned* 
Z- Par t i ci oa t \ y e pi ann i ng of workshops . The workshops are planned 
participattvely in terms of the needs of the group at a particular 
historical time in the life uf their development program. The workshop 
for wHKh the* organwers hod prepared themselves fs locally re-invented 
by the participants. 



1' P£Til£l lMi jii. cont rol con tent, emphasis and schedu le. The content of 
teaching- learning i the emphasis given to various topics and the schedule 
of activities are controlled by the participants. The workshop works 
according to a rolling plan — everydayi the program for the rest of the 
woikUiop is reptanned. Schedule for the day is prfntetf at the end of the 



4 - £ learning cginiunity . The participants became members of a learnirf} 
ctmuunity who llvei work and play together. Everybody lives on the same 
location using - the same hvel of fac-fVities, Late arrivals and early 
departure* are not allowe*:. 

^ f j£|i 1 li£ 0 - r ^^ not teachers . The orjanizers look at themselves as 
facilitators lather than teachers. Contributions are encourajed from 



day. 
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everyone. Teaching-teaming methods Include lectures* discussions* panels^ 
group work, and individual tiHoringi depending uPon the needs, 
G, Xhejctjon orj eny itlon , taphasts Is on relating theory to practice; 
to enable participants to do soroethirki practi cal^-wlth thei** Ideas, During 
the workshop, ea* participant is asked to prepare a proposal for an 
evaluation studj^ This evaluation study must be implemented during the 
year between the workshop they are then attending and the next workshop 
at which they must, report on the completed study, 

7, Con tinuou s evalu ation and feedba ck. The workshops are continuously 
evaluated by using formal instruments as well as H the daily steering 
committee meetings every night where drop-ins are not only allowed hut 
encouraged, 

*** The l n A^ raUjrt rol e of documentation . Special documentation is 
prepared for each workshop- First, this docunentatfon assures that ito 
-lectu re^ notes need to be Prepa red^ and distriButed^at thej*ar,?i£hop — A<1 — 
the time available can be spent on learning, A1so» it previous the 
trainees with the big picture of the area of evaluation to onsure that 
participants do not confuse p»rt with the whole. What they may have 
decided together to study during the workshop may not be the whole of the 
evaluation process and of the evaluation knowledge, 
9 , I ns t Hut i o na Hjatj on _ f o M o ca 1 l_tr aMf er_ of r e sjw ns i b H j ty , 1 here i s 
emphasis on the caPacftatlon of local professional personnel and on the 
building of institutional capacity for the trans/er of responsibility, 
local faculties are associated vfth the organization of workshops and 
given time and provided resources to assimc leadership. An Educational 
Evaluation Resources Conmfttee* for example^ was established in Kenya as a 
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step towards ¥ the institutionalization of the evaluation capacity in the 
own try. During ldd^-86, the host institution for the DSE workshop 
will indeed be on its own in offering evaluation workshops to trainers of 
development workers and change agents. * ( 

Oeve loping Descriptions of Training Programs 
in i Design T enia 

in the opening paragraph of this chapter* we had suggested that 

evaluators must be able to understand what they seek to evaluate and must 

# 

be able to develop adequate descriptions of the phenomena to be evaluated. 

Since the purpose of evaluating training programs, typically, wilt he tu 

intervene in the training process to improve it, these descriptions of 

development training programs must be in design terms. * 

Jo jiejie 1 o^-s uc l> descriptions we must once again engage in system 

thinking and return to the model of development training design and 

evaluation included earlier in this chapter. This time we should use the 

four parameters of systems (inputs* processes, contexts antf outputs) to 

list the variables involved within the universe of development training. 

The chart on the following page provides a first list of variables 

P 

that will typically appear under the four parameters (inputs* processes, 
contexts, outputs) in the case of each of the three systems (training 
sysUm t performance system, and social system) that appear in the model 
referred to above* Can you think of any other variables? 
/ 
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INPUTS 


PROCESSES 


CONTEXTS 


OUTPUTS 




A. Trainers 

3, Trainees 

C. Materials and 
facilities 

r 


A, * Instructional/ 

formative 
6, Organizational/ 

structural * 
C. Oistrfbutive/ 

ma intend nce- 

rel a ted 


A. Organizational 

B* Environmental 
( extra - 

organizational) 


A. Trained development 
workers 

B. Experienced 
trainers 

C. Training research 
and training 
materials 

0* Unanticipated 
results 


— 1 

-< 33 

if 


A. Trained field 

workers 
8. Learners in 

conwiuni ties 
C- learning materials 

and facilities 
0. Economic inputs 
; E. Inscitutional 

inputs 


A. Planning and 

needs assessment 
8, Organisational/ 

coordination 
C, Animation/ 

motivational 
*D. Instructional/ 

socialisation 
r E. Distributive 


A, Organizational 

B, Communal 

C, Ethnic/ 
linguistic 

0. Rurah/urban 

E, Socio-economic 

V 


A* Informed and 

resociatized youth 

and adults 
B» New institutions 
C* Developed and 

self-reliant 

communities 
0, Unanticipated 

results 


33 

— 1 

n O 

*i 

s 

-n 


A, National mission 

B, Leadership/ 
manpower 

C> Informed citizenry 
0, Institutions 
t. Technology/ 

material 

resources 


A. Ideological/ 
Pol icy making/ 
Planning 

B. Legislative/ 
organizational 

C. * Communication/ 

mobilizational 
0- tnffastrvctural/ 
distributive 


A, international/ 
world system 

B. Cultural 

; C, Political/ 

socio-economic 


A, A developed and 
just society 

B. Unanticipated 
results 


-< Q 
V) o 
— i — i 
n O 

O 



Figure 2. Variables under each of the four system parameters of the training, performance* and social 
systems. 



Ijitf jrajnorV ^rilvers ti and jjie tra iners' tasks 

The generic model of development training design presented earlier* 
together with the chart of parameters and variables that has followed the 
model should enable us to chart the universe of training In the area of 
development. It should help us see the variables that will enter the 
training system and to see the variables of the performance systaii and of 
the social system that will directly Influence training design and imple- 
mentation. 

In the next chapter we will deal with the issue of generating evalua- 
tiun Questions and discuss the criteria for choice ajiiontf such, questions^ 
to be dble to develop useful evaluation agendas for evaluating the 
effectiveness of development training. 



To be able to cvoludte development training, we must understand ttic 
process of training design While trailing design has a long tradition of 



theory and practice In the army and industry, the development training in 
the Third World settings raises some unique problems^ Development 
training wu>t often involve more than capacltation for the role* it must 
dhu involve "role invention" <md H ro1e socialization. H A model for the 
dusign uf development training programs is presented which can also be tfted 
for adequate descriptions of ongct ig training programs. 

Things to Do or Think About 

1. Sketch the main features of the training model or approach in use In 
your training center or progiam (as we have done In this chapter for the 
Unusio's Operational Seminars, the Participatory Training Model and the 
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Action Training Model). * - 

2. Using the figure on page 51* analyze the Input-procdss-context- 

output variable In the case of your training program. 
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CHAPTIK V 

ASKIMi EVALUATION (JUESTJONS, MAKING 

LVALUATIOH ACitNOAS * AND TYPICAL EVALUATION Tll£H£S 

There will be situations when Uie trainer-evaluater will have 
evaluation question given to hlnfor her, Ideally, however, it should be 
possible foi the tra mer-evaluater to go through the systematic process uf 
describing the total system of training* be aware of the options typically 
avuilable under each of Che input, process, context and output variables 
operative within the training system* on the basis of problems experienced 
or the potential visualized, generate a set of evaluation Questions; and 
choose fran amung these evaluation questions, those that must be answered 
on a priority basis, 

\ !!£. f trii Stefij^ Description 

In Chapter IV* we nave already presented our approach to developing 
des cr iptiuns of training systems, A wodel of development training design 
and evaluation *as offered, followed by a chart listing all the variables 
under the foor system parameters {inputs, processes* contexts and outputs) 
fur each of the three system (training system, performance systau and 
socio-political systun) in interface^ th each other, * 

An outline cli<*rt as shown on the next page should be used to develop a 

description of the training syotaii In which the trafner-evaluator is 

involved. Woi e attention should be paid to the row dealing with the 

"training systuu," The otner two rows dealing with the "performance 

syston" and thd 'socio-political systuu" cuuld receive less exhaustive 

* attention. Ihe entries in the cells of the chart should not be general 
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Figure 1. An outline chart For use in developing a description of the training system In which the 
trainer^evaluator is involved. 
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but should inUude specific and concrete statements of the realities with In 
the training system being described. 

S tg|> Twoa Know What the Possible Options Are 

Trainers-eva luators should not only know what specific variables enter 
into the conciete situation of their training program but also know what 
options are typically available under these variables. As students of 
irajnind design they sitoutd know "what Is*" as well as "what can be." 
White it is often possible for trainers to Influence the choice of options 
within tin* pei furiitifnc system and maybe even in the larger socio-political 
system, in the following table we list and discuss opt inns available within 
the training system only: 



PAKAMLTtKS / 
Variables 



Options 



Uiptm 

Trainers 



ideational levels and knowledge of subject; 
their social class and value orientations 

txtent of field work experience 

level of CQMiituent to development work 

Teaching competence did teaching experience 

Direct appointment versos secondment f^un ** parent 
department 

Continuity versus turnover 
Work loads of trainers 
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Trainees Educational background 

Pre-servlce versus In-service groups 
Social class and value orientations 
Comnl tment to development work 
Motivation to learn 

Recruitment modes (deputed, sponsored, oPen 
competition) 



Teaching Teaching materials—quantity, diversity* quai ty 

Materials and 

Facilities Indigenous versus Imported Instructional materials 

Instructional and duplication equipment 
Physical Plant of tlie training center 
Transportation for field visits 
Budget 

Level of program commitments 

Access to experimental villages and pilot 
Projects, etc. 



PROCESSES 



Instructional/ 
format lonal 



Conceptualization of training as knowledge 
transfer, sktlk training, behavior ntocMMCfltlon, 
socialization, etc. 

Integrated versus discipline-oriented curriculum 
development (I.e., Instructional organization) 

lask-sPeclfIc versus training for core skills 
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Teaching and learning styles 

Substantive knowledge versus process emphasis 

Presence versus absence of currlculuu validation 
through needs assessment 

Conceptual versus Held operational orientation 

Availability or nonavailability of counseling and 
guidance services 

Instructional delivery — face-lo-f ace. distance* 
etc. 



Organizational/ Organizational Health status 
structural 

Organizational capacity rating 



Distributive/ Quality of administrative support 
maintenance- 
related Residential facilities for staff and faculty 
and for trainees 



contexts 

Organizational Organizational culture 

Institutional relationships (horizontal and 
vertical) with other organizations 
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Environmental Surroundings (Closeness to a bar versus a 
"retreat" situation) 

General social climate In the country 



0U1PUTS 

Trained development workers 
Emergent role Identities 



Experienced trainers 



Training manuals 
Radio programs 
Correspondence course? 



New professional stratifications 
Contributions to a "brain drain" 



Stop 1hree: p Genera tljigjvajuajjjon Questions 

Ihe tralner-evalua tor must now confront "Mutt Ts* with "What can be." 
fhe tralner-evaluatot should now look bactc critically on hh or her Hay-to- 
day exhericnecs within the training program and try to articulate ctearly 



the problems which were there but perhaps were harJ to yet hold of, Iho 
trdlner-evaluator should also look at the existing training system Positively 
diU think of the higher returns that could be obtained hy waking Sume changes. 

In all these cases, the tralner-evalua tor should be able to state some 
information need: We have lite problem "X", but we do not have the 
intonation "¥**. Ori if we had the information H Y , we could take the 
prainsliHj step "X*\ with confidence, 

luu Important points must he mentioned here, 
' • A distinc tion shou ld be made between evalu ation problems and adai nistr a- 
Live nrublejtis. To administer Is to direct and superintend the execution* or 
conduct of ** p*oyrd«i. If administrators, for reasons of incompetence or for 
laiA of responsibility fail to direct and superintend a program, the 
problem is administrative, not that of evaluation. Evaluation can only 
assist dfkiiustiation by providing needed feedback data and by testing 
VdMuus program assumptions* It is nut a substitute for good administration, 
^ + Lil!iiilliL n , r equire ana l ytica l ev aluation or collection of fr amewo rk 
d£tg, Ihere will be evaluation questions winch will have to be answered 
through conceptual and uperatfonal analysis rather than by going to the 
field iur cullecting data. Is the participatory metluKl claimed to used 
in a training program* actually employed in the training protocols? Is Die 
integrated con icului. concept actually embedJed into the training plans* 
trailing mateualsi and training delivery and schedules? These Hnestions 
reijuiie analytical answers and not necessarily collection of data, Again* 
Policy documents* the nation's five-year economic plans and census data may 
have to be used fu building a framework to the evaluatiun of an aspect of 
the development training program* 
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Step Jour^ JFr o^yajuatlon Q uestions to E v aJuatii >n^flgend_as 

The evaluation questions generated In Step Three above may all be ' 
Interesting and promising bat It may not be possible to answer all of then 
In the particular context of a training program and within the resources 
available, hi such a situation, a particular evaluation agenda must be 
made, to be followed within a particular time period. 

The following criteria enfght be useful In the choice of evaluation 
questions for Indus ton In the evaluation agenda; 
I, Availability of design options 
*l. Significance of the evaluation question 
J, feasibility of faplanenting the evaluation stmfy, 

O) 4val TabH j>f options. All of the variables entering * training 
situation may not be under the control of the training specialist* In 
other words, the trainer may not be able to change the values of the vari- 
ables In any significant way. If such Is the case and training variables 
are intnutable. It is no use evaluating th^nt because they do not offer 
design options anyway. 

f^J Sfjnfjlcance of jlie evaluation jjuesLlon, If a variable does ofter a 
design option. It will make sense to evaluate It. If fn a relative sense* 
ft offers a significant option. The significance has to be in terms of 
results In the effectiveness of training or in its efficiency- m either 
case, ihe returns fnni the evaluation effort should be worth the effort, 

(3} Feaslhjllty In regard to. available resources. The evaluation question 
chosen ami the evaluation design that is necessary for conducting the 
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evaluation should be within the capacity of the training center or 
institution. Reasonable mtount of resources should' be available for 



Typ ical Evaluation Thanes in 
Oevejpjgiient J j 'AifllJl^ 

The central function of evaluation of training is obviously to 
validate the assumptions on which training design has been based. To .put it 
differently, trainers engage in evaluation to judge the worthwhl leness of 
their training programs* in terms of some criteria* on the basis of 
information that can be-collected or is available. 

The bask aims of the evaluation of development training can be stated 
as fol lows : 

h measuring the abilities* skills and value orientations acquired by 

trainees during the various stages of a training cycle or at* the end of 

training^ and* later* to measure performance on the job in actual field 

situations, , 

I. on the basis of the above* pruviding feedback toVainers in regard 

to choice of training objectives, content, methods* materials 

instructional settings for appropriate remedial actions, and 

3. providing information to policy makers and planners in regard to the 

overall training polkies, designs, patterns of recruitment* role 

definitions, performance *i\ the field by role incumbents * institutional 

settings of training and the total allocation of resources to the training 




effort* 



These basic aims of the evaluation of development training wfll 
typically appear as the following evaluation themes: * 
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K B£se-lfne survey * 
Needs assessment 

3. Curriculum evaluation _ _ _ 

4, Achievement and attitude testing 

5* Institutional or organizational evaluation 
k, Cost-effectiveness evaluation 

7, Evaluation of Impact 

8, Management Information system* and 

9, Monitoring and quick appraisal* 

1 * Ba se-l ine s u rvey "~ 

Base-line surveys of communities are undertaken to establish the 
economic* social and cultural base-line against which later changes can 
be measured. Community development workers generally would conduct extensive 

base-lfne surveys In rwwiwHHPc they *t>*k fo s^rvp. HhetfiYec pnc<Hhi<» t 

tr a tners-e valuators should use already available base-Tine data to design 
their training programs tor development workers and change agents* Jt Is 
possible* however, that the base-line survey already conducted had not 
anticipated the special Information needs of tralners-evaluators* 

The special Information needs of tr a Iners-e valuators may deal with 
(1) role considerations* and (2) knowledge considerations, A tralner- 
e valuator preparing family health education workers* for example, would 
need to know the current child-rearing and health practices within 
communities, level of knowledge of nutrition* lack or otherwise of hare 
gardening; and level of consumption of animal proteins, At the same time* 
tlie tralner-evaluator would be Interested In how this knowledge Is currently 
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acquired by mothers^ whether traditional educational roltfs exist that 
disseminate this information; what other more modern -secular roles have 
already been introduced within those coemunities by the government; and 
what expectations one should have about the introduction of a new rule of 
the family health education worker, 1 

As can be surmised, trainers in cooperative training colleges ur 
IrSiiierT of agntulturdl extension workers will have to def ign -base-1 ine 
surveys tu fit their special information needs about existing economic 
knowledge* behaviors and structures; and the design of roles of extension 



2. Heeds assessmen t 
f- 

Trainers-evaluators may have to conduct needs analyses at various 
systemic level*. They may conducjt a general needs analysis at the national 

levelHo^W&t^hose^ed^n-jj^niii^^ also do a 

needs analysis at the performance system level to see what demands a 
particular program or project will make on development workers and change 
agents. Finally, needs assessment may be conducted within the training 
sy stair itself to design appropriate knowledge, value and performance 
content of a training course or workshop* 

A guod needs assessment* typically, will cover all the constituencies 
involved within a training system trainees, their field supervisor, 
dt^ulopiient aJuiinisti ators * cewjuunity leaders* and uiiuiunitics themselves* 
The final training design should be done on the basis uf the various needs 
proMlus genernted by these different constituencies and groups. 



agents. 
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3- Curj Icj^lutn eva luation 

[valuation of training currlcitlim will be a frequent evaluating tlieme 

of tralners-evaluators. They may need to evaluate particular Items of 

Instructional materials — a primer, a handbook* a set of charts, a 

simulation-game. They may want to evaluate a particular training method, 

for example* team facilitation versus single tutor. Different systems of 

program delivery may be tested: correspondence courses versus night 

schools; teaching mothers or teaching families, etc. Finally* the overall 
* 

effectiveness of a training curriculum may be the concern of tralners- 
evaluators. 

A, Achievement and .aUU jg te test 1 n tj 

, A considerable part of evaluation within a training setting wit I 
consist of testing. It will be testing of trainees as they enter the 
training system and their testing as they leave. This testing will cover 
knowledge; diagnostic and performance skills; motivations, attitudes and 
values; and ccntnunlcatlon and process skills. 

Some of this testing may have to be done not on the trafnees hut on 
Individuals in the communities* to be able to Judge change agents' per- 
formance by the fmpact they may have had In the ccnnitml ties . 

5. Institutional or^ organ \i at 1 ona 1 m eva \\ ua tjpn 

The quality of Institutions or Organizations determines the quality 
of services the^e organizations will be able to Produce and deliver. 
Unloi imattfly. *ery little attention seems to have been paid by development 
ti diners tor by development specialists qenerally) to organizational 
traits, In^t i ti/tions o» Organizations can be studied along two General 
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dimensions, []) organizational climate* and (2) organizational capacity. 
Organizational capacity Is determined thrash an accounting of an 
orgamzati oil's resources In relation to Us mission. Organizational 
climate is a conceptualization of an organization's social life -- 
mender's identification with the organization and their satisfaction or 
dissatisfaction with organization's derision making style and patterns. 

6. Cost-effectiven es s evaluati on 

Iwo tenns are in use m the literature of evaluation In the develop- 
ment sector, cost-benefit analysis and cost-effectiveness analysis, both 
these analyses involve comparisons of costs and outcomes* but the nature 
of comparisons differs. 

Cost-benefit analysis is possible fahen outcomes can be given clear 
economic values in dollars and cents. This soft of economic analysis is 
seldun possible in education and extension where non-material effects are 
Hie most significant but can not be assigned material values. 

Cost-effectiveness analysis is used where outcomes can not he 
expressed in monetary terms because of the absence of market prices for 
outcomes. 

Therefore the levels of outcomes themselves are compared In proportion 
to the costs incurred in each different case, 

U may be also useful ,for trainers-evaluators to work out unit costs, 
i,c\> costs per trainee trainedt to be able to allocate resources more 
eflectively, 
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, 7. E_vjSljjatJioii if o/ fc Jupact 

The study of the Impact -of training on change agents must go beyond 
testing of trainees within the organizational setting. Tralners-e valuators 
must go to the conrmunitfes where their trainees work. Their questions 
must, however, be sharply focussed: Old the change agen role fit Into 
tlic social setting? Was the role performer able to teach* demonstrate and 
resoclaMze? These questions wi 11 he Possible to answer if base-line 
data was collect d earlier. 

The study of th* Impact of change agent roles on communities must 
provide proper time for the new role to go through the period of adaptation 
and use by the communities. They should have time to relate* learn and 
adcPt. Such "^tertcy periods** may have to be many- months (If not many 
years) long. 

Also.* In the study of the Impact of new roles and new teaching 
within communities* trafners-evalvators should Took for botli F the anticipated 
and the unanticipated consequences of the introduction of charige agents. 
Have role conflicts onerged In relate to traditional roUs? is a new 
qroup of power holders emerging within, communities because of new role 
performers? Mas the change agent brought in bureaucratization; resulting 
^n the destruction of local lnftletlves? - u 

H, Management information sysijfl* IffJSJ 

Tralners-evaioators can not depend on special evaluation studies for 
all their information needs. Special evaluation studies »tafce time and 
resources, while decision-makers (Planners, administrators, trainers) need 
Infestation teqularl^ to make day-to-day decisions. For fulfilling these 
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tidily needs uf information! tra iners-evaluators must build manayaitent 
Informal lun systems (MIS's). An mi s is developed by die systematic colle*_tii 
'and. storage of data that » generated by a program in the very process of 
Us in)plta<teiitation. All that is required Is that data recording devk» 
s,udi 45 registers* application forms, grade books t diariesi log booU^ 
^ supervision report funus oe devised and their use be mandated. Supervision 
iepoits and periodical reports be regularly obtained and filedt a* also 
instructional materials, and related records and documents* 

Sjdi data should be processed regularly at various decision points to 
ituU* appropriate management decisions. l\ should also be used for secondary 
analysis <*s part of special evaluation studies, In other wordSt special 
evaluation studies shuold be able to use* whenever possible the data 
already stored In Uie MIS* 

9. y£>iliwlni l n i! gunk appra isal 

— . _ ^1 

In Lhapter I, we made a distinction between evaluation arid monitoring, 
lite dictionary meanings of monitoring are to w?.tch» observe, check and 
sometimes adjust. Monitoring is thus a term used for collecting the 
status information from tlie operating and implementation levels uf a 
pi am tu deteimfne performance in comparison to expectations. It should 
be unviou* that a good ^ifs will help the nitni toring process. 

Another term coning in use mi evaluation literature is "quick 
appidisal." ifuuk appiaasal is tlie child of necessity- Systematic evalua- 
tion take:* time, yet in wany instances evaluation needs may be both urgent 
and compelling. A ^uiU appraisal wMl typically be done by a visiting 
team of e/aloatoii. ubiiuj bind 1 1 satisfies and short tests and schedules, with 
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findings Provided to decisionmakers within four to Six weeks* These 
findings will be in the form of strengths and weaknesses In the program; 
factors that seem to *>e responsible for those strengths and weaknesses; 
and what corrective actions seen clearly called™ for. 

Suimrary - 

lo ask good evaluation questions and to make good evaluation agendas* 
trainers-evaluatorS must engage In "system thinking* 1 and begin with 
descriptions of training programs for 'which they are working. They should 
net confuse atfolnlstrative problens with evaluation problems, in choosing 
evaluation questions for study* the criteria used should be (l) the 
availability of options to make changes In the training system on the 
basis of evaluation results; iz) the significance 0 f the question fn terms 
of retur" *Jie time and resources conmftted to the evaluation study; and 
(l) the feasibility df conducting the evaluation Study. Typical evaluation 
thones In which trainers-evaluators will be Interested are base-line 
surveys, needs assessments* curriculum evaluatloo* achievement and attitude 
testing* Institutional or organizational eva Ion* cost-effectiveness 
evaluation* evaluation of Impact* development of management Information 
systems and moottorlng and quick appraisals. 

Ihlngs to Uo^or Think ,JJfeP«t 

1, What are some of the options available In the context of your training 
pioqrmn regarding recruitment of trainees* teaching methods and materials* 
or location and delivery of training? What arc some of the Information 
needs for yon to make sensible choices among available options? State 
Uie<n? Information needs lo the fptm of evaluation questions. 
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i> What do you need to know about the most significant # dtsuands made upon 
development workers and diar^e agents by the "perfohuance Systeai" in your 
itfte? Reflect this need for information and knowledge in your evaluation 
questions * 

L What die some uf the important facts of socio-economU and political 
life in yuui environment that must be reflected first in training design 
ditdl then in the statement of evaluation questions? 
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CHAPTER VI 
l 

URfflttG -FORMAL iWOSALS full 
EVALUATION STUDIES 

Successful, cast-effective and timely completion of an evaluation 
study requires considerable forethought and pre-planning. This thinking 
and pre-planning can be best done within the framework of developing a 
"formal" proposal for the evaluation study. The process of developing a 
Proposal for the evaluation study can be used to systematize tlie evaluator's 
own thinking; to clarify technical, secretarial and material needs of the 
study; to take stock of available resources* to request and receive 
consultant help, if necessary, on various aspects of the evaluation study; 
and to use the Proposal as a tooT of communication with administrators and 
Interested parties. 

Proposals for evaluation studies dealing with the effectiveness of 
development training fn the various development sartors* sirggesti 
should Include the following elements: 
I. fhe developmental context of the training program* 

Z. The description of the training program in design terms. 

*■ 

3. Hie larger "complex of problems 1 ' within the total training Program. 
1. fhe evaluation Problem chosen for study. 

5. Justification for the choice of the evaluation problem chosen for 
Sjtndy 

6. A review of available knowledge directly or Indirectly related to the 
evaluation problem chosen for study. 

7. fhf? finest Ions and sub-questions to be answered by the evaluation study. 

7J 
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a. The evaluation *nudel<s ) or approaches) chosen for the evaluation 
>tudy , 

9. Evaluation design (ur steps and procedures) to be used in conducting 
the evaluation study. 

10. Instruments and tools to be used for data collection. 

11. kians for library and documentary research, arid plans for field work 
wiei-e collection of field data is involved. 

1^. Plans for analysis of data and Preparation of graphics, if needed, 
li. UutHJfetdry plans. 

14* Reporting plans: who Lo report to. when* and in what form? 
lb. inbhograPUy . resource persons to be 'used, etc, 

i 

A ut*jiiirier> wnnmj his or her first proposal for an evaluation 
study* way find it useful to go through steps 1 to IS. mof^or^less . in 
the Ofdur given above. The to ore experienced proposal writer may be able 
to jtuip buck and forth to various steps; frou step 4 to step 7, to step 
]U. tu step 12 and so on. Again, in the settings of workshop and seminars 
uf it^r t durations, it may be necessary Lo focus on suite steps and not on 
othci>. 

It ihuuUt also be kept in mind that until the final propusal is ready, 
the hjMous pax U yt the proposal will require constant review and revision 
The^ development ot tools and tnstiuiients may require a look back at the 
indicators ihusen lor the study. A review of the indicators may require 
revjrUiuy of the ti/u luaUori Question and of its Justification* Even a/,ter 
thu, piupusal is all dune, the realities of the field may demand changes 
and iivisiuus, once a-jam. One should be mentally teddy for these never 
&ndw>} revitfws. 
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He wiH now elaborate" and expand up*n the various effluents iisi^d 
above* citing examples where necessary, 

1 . The de ve j opn e i n_ taj _cq nt ex t 

Development requires learning of new knowledge, new skills and new 
attitudes by workers, fa?<roerS and housewives. To deliver this learning 
to those who need it, we need trained change agents. The role and func- 
tfnns of the training institute or training program of interest to the 
e valuator should be put within the development context. The training 
program's contribution to the national effort !n the training of manpower 
for development should be clearly brought out. 

A statement of training objectives shoufd foliow. If the institution 
offers a variety of training programs, each different Program shovid be 
fisted* with specific objectives of each program indicated separately. In 
sane cases. It may be useful to include the organizational char.t of the 
training institution or program. (See Chapters 11 and HI.) 

2* Ihg desj:Mj_t_ign of_ j he training program in des ign terms 

A description of the training program In design terms must be 
developed. First, the general characteristics of the training appr'jth 
should be listed' 

(a) Is the training supposed to be general or specialized? 

(b) Hoes ft emphasize teaching of critical consciousness or the 
dissemination of knowledge and sfcfUs? 

(<J Is the training planned participative^ or Is it formal and pre- 
packaged? 

(d) Is the training offered academic or operational? Aral 

i O 
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(e) boes elite ttaidiuy seek to teach entrepreneurial values or communal 
values? 

Inere may be *ome other important questions that could be asked, tut 
the above list should provide a good startiny point. 

These general questions about training design pmst be followed by a 
desuiptioii of the training urogram to be evaluated In s ysla ii ter ms ■ . The 
fuui ayU&i) parameter* (inputs, processes, contexts and outputs) shoold be 
used to describe the training system In concrete teiins and vai-es. (See 
thdptets Itft especially trie model of development training design and 
evj Nation on page J9, and the chart including variables under each of 
the four »j/stui» parajiielurs of the training performance and social systops 
on pfl'jc &|.) 

Ivatuation problems arise from a lack of information or a lack of 
understating, He otay have nu information or insufficient information on 
inpu^u arid about tho context of our work, We may nave less than adequate 
urtdor :>Uf*di*kj of thu processes and their application within our particular 
:>ettir*j, He may have j»o measure of the quantity or quality of our outputs. 
Ihe>e >Uot UiHttUhj* tuyuther wi 1 1 create a whole "complex of probities" in any 
tjau.iuy piuv'* 1 "- Hide***, a tTaunwj utstitut iuji 01 a training program is 
unlikely uver to bu short of evaluation problems. 

In devuTopitnj 4 proposal fur an evaluation studyt an evatualor should 
*evicw lite uhule itt ot inter j elated problems found to be bothersuiie to 
uroyidH* admifuitr aloi :> fl nd decis lun makers The evaluator 'imst* however t 
di»tt^jui»N between evaluation problems and purely administrative problems, 
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Evaluation Problens arise fron lack of Information and understanding* 
whereas administrative problems arise fron Incompetence or deliberate 
neglect of duty* Administrative Problems can not be solved by 
evaluation. (5ee Chapter V.) * £ 

^ * ^Vil^Ai L°i ■Pr.QfeJjp ch osen for stud y 

The evaluation Problem chosen 'or study will have *n he not* nut nf 
the complex of Problems described under the Preceding section, "ihe 
Problem Complex.* 1 

A good problem statement Is one that is as concrete and specific 
as possible: 

In Place of the total training effort of an Institution or program* 
it may be Preferable to evaluate a specific Part of the training effort, 
in place of all aspects of a training effort* it may be preferable 

■Cr 

to evaluate only seme asPectsof a training effort. 

»tmay be Preferable to cover a sa^Ple" of a population rather than 
the total universe. 

it may be Preferable to study the implementation of a tra1n£n£ program 
during a specified time penoo rather than over the total life of the 
program. rj 

it may be preferable to loofc for specific and concrete effects of a 
training effort rather than Its broad and generallzad Impact. 

We are not suggesting that It is Impossible or undesirable to study 
the broad Impact of large scale training programs In terms of their general 
and long-term influences on large groups of trainees. All we are suggesting 
Is that* In most situations, It might be more useful to bn specif it rather 



ERLC 



/ t 



m 76 
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than gen<u*al. 

Urieiher the evaluation problem Is defined in genera) or specific 
terms, ambiguity is not permissible under any circumstances. The evaluator* 
in stating his or-het L evaluation probluiu should be most careful aith words. 
The wuids snuufd mean exactly. what is in the in fnd of the evaluator* nothf >>y 
mure and nuUnwj less, leaving no scope tor alternative interpretations. 
(See Cnapter IX below.) 

J *iSiify.L<ia choice of the evaluation pro blem 

The^chotce of one evaluation problem fran a total "canpte* of problens" 
cad nut be capricious and arbitrary* The evaluator should ae able to 
Justify hi? or her choice of the particular evaluation problem 

Tne Justifications may range frcm the political, to the programmatic* 
tu thu merely possible. An evaluation problem may be justified because 
the donors want it studied or because the planning department or the 
president's office needs to be pacified. At other times, the evaluation 
problem Uiusen way have important policy Implications or way produce 
crucial feedback absolutely necessary for the future Planning uf a program. 
Ui an evaluatiun problem may be Justified In terms of feasibility 
suiietiiiny that can be accomplished with the minim uu of resources even 
thuuijii Ihere might be other more important evaluation questions which 
snoolJ Nave been tackled first if resources were available. (See 
LnapLer V. ) 

tp levies* of available research ^j^^Agei'iejice 

available theory jriO research may help an evaluator to define and to 
clarrry the evaluation problem and help in asking the right questions or 
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frame the right hypotheses. Other evahrators. In other training settings. 



administrators and trainers who have worked long In similar training 
situations. An attempt should be made to collect available knowledge, 
experience and opinion as Part of developing the evaluation proposal. We 
should learn from other people's experience and shohiu not waste our lives 
in reinventing tlie wheel! 

7, As Jt1r<j .questions a Jid_su b - que st jpnj. ♦ 

ft Is Important to translate the evaluation problem Into a set of 
questions to be answered or hypotheses to be tested. Such Questions may 
Involve mere countin g (How many of the literacy teachers tralned.durlwj the 
last S years are still teaching literacy classes?)* or describing (tfhat Is 
the nature of the emergent role of the family health ffeld edocator In 
Hakuru7). Hypotheses wl 1 1 Involve relaUji (fralnlng In cooperatives 
management Is positively related to effective Performance in the field); 
°*" fw dlAtJng (Elementary .school attendance In communities will go up as 
rates of adult literacy go up In those caiimunf ties). Questions can be 
stated as hypotheses and vice versa. One need not. however, state nne's 
evaluation Interests both as questions and hypotheses* at the same one 
time* [hat will be a useless redundancy* It mfght be best to work with 
questions and sub-questions and leave hypotheses alone* 

8* Eva [Nation mode I (s } and appjroadi (m ]_t£ be used 

Evatuators should give careful attention to the choice of evntifat'on 
models and approaches, At one level, the evaluator can choose hetween 
* the classical (also called the scientific) Paradigm or the naturalistic 
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paradigm uf evaluation and research. Ihe choice between one or the other 
paradigm will le^d later to different kinds of evaluation modtls and 
infomutioii-gathering approaches and techniques. (See Chapter VI J J be*.ow,) 

9* £ v ^Ijj^U<>!Ljks i!jn or steps and procedures 

To have an evaluation design means to do all that is necessary to 
de and the conclusions of your study from attacks on validity and 
reliability. Quite often, the design may involve nothing wore than the 
choice of riijht samples. At other times, evaluators may have, to have 
base-line data or to have matched or control groups of some kind. Major 
steps in the conduct.of the evaluation study and the procedures to he 
followed at e ^ch step should he outlined In this section of this proposal. 

In the case of naturalistic evaluations* the problems of design take 
a different form and will he discussed in Chapter XIL 

10, j nstrunents and yjojs of data collection 

ihe proposal for an evaluation study should include a discussion of 
the touts and instruments that will be used for the collection of data. 
Preferably the first drafts of the tools and Instruments should he attached 
to the proposal. 

hiere are two prior questions that the evaluator must face before 
tjuttiiKj on with the construction of the tools and instruments: (l) What 
is the. unit of analys b ? Jn other words, wliere are effects and 
conse^eence^ likely to appear, in individuals* in families or groups* in 
ur^aiiiiations^ or tawunities? (2) what will be the iiidj cators of effects 
oikl cotiieijuyncyi haviiitj actually appeared? In other words* what responses 
and bdiaviors* for eAtwple* will indicate UianSe in motivations or In the 
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learning of self-reliance? (See Chapter IX.) 

The most frequently used tools and Instruments In educational evalua- 



tion are: 



Tests 
Interviews 
Questionnaires, and 

Observation schedules* * 

Tests can perform a variety of evaluative functions, Achi evement tests 
are like examinations to which we must have all been exposed during our 
years of schooling. Achievement tests measure achievement of trainees on 
the subject matter taught to them. 

Aptitude tests measure the natural er acquired bent of mind. They are 
futuristic. They tell you whether a trainee Is likely to make a good field 
worfcer or Is he or she most likely to prefer a desfc job? These tests are 
much more difficult to make than achievement tests. 

Achievement tests mayibe pencil and Paper tests or they may seek to 
measure actual Performance on a task, When performance Is the focus of 
measurement, these may be called performance tests . Performance tests are 
much more difficult to score than paper and pencil tests. 

Finally, ^valuators should understand the distinction between 
standardized, tests and crJterlon-referenced tests . Standardized tests are 
typically used within formal Instructional settings. Evaluators In 
developmental settings wit! most often bo dealing with criterion-referenced 
tests: tests which determine und evaluate knowledge or performance 1n 
reference to criteria of success established by the trainer In relation to 
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me particular trainee group* In the particular training setting. 

Jj iter views may De structured (where you ask pre-determined questions* 
using exactly tfie worJs in wnich your questions Itave been written); or 
they may be unstructured (wherein questions ar e used merely to start a 
dialog* and further questions are framed and raised by the Interviewer 
a5 he or she yoei. along probing and encourayiiiQ). Unstructured interviews 
ai'e the tools ot naturalistic evaluation. 

He can have Individual interviews or group interviews. Me can use a 
fomi tu record interviewee responses or a note book to record responses* 
we can use a UPe recorder; or we may want to write a note on the inter- 
view only after the interview. (Sane evaluators do this so as not to 
inhibit the natural flow of interactions between the Interviewer and the 
Interviewee). 

btruclured interviews! typically* y se predesigned forms. Each ques- 
tion is followed by same space m which the interviewer can record the 
answer, Jf the interviewee (rather than the interviewer) records the 
responses* we c«jn call this forui» a gugs tionnaire , Questionnaires can be 
filled in person- to- person situations to have control on data collection. 
The evaluator con ensure that subjects sit down to till in the question- 
naires and the evaluator can take the responses back to the office. 
(Juestiuimairesr however, can be very expensive to administer in face-to- 
face itttmysi if the respondents are spread all ovi?r a district, a 
province or a country, In that case Such questionnaires rouSt be mailed. 
Responses trun tiiailtd questionnaires seldun. if ever, show hundred pur 
cent returns. 
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In some,* etiraluatfon situations, important data woy have to be 

y 

collected through obs erva 11 on — by bofng there and watching and listening. 

Observation, aga1n t as lo the case of Interviews may be structured or 

unstructured. Structured observations will, of course* require pre- 

designed observation schedules Unstructured observations may be recorded 

1i- a note book or spoken into a tape reenrder by the observer* Observations 
i 

may be conducted by a single ^valuator or by a team of evaluators. fojaln* 
observations may be done at a single tim& ( or serially* by returning to 
the same sl^tlon at different Intervals of time. We can even apPty 
sailing techniques to observation by making scmpi^s* random or otherwise, 
of places to be visited, of days U visitation and of time nntts when 
observation wtM be conducted. 

Observations way sometimes be done obstruslvely (subjects of 
evaluation wftl know that they are being observed) or unobstrirtlvely 
(subjects of evaluation will be unaware of being observed). This late" 
type o( observation Is the specfal tool of naturalistic evaluation. Even 
while doing unobstnisi *e ob$e r ?at1ons » one could be a detached observer or 
a participant observer. 

JEvaTuators have to be careful about the choice of tools and Instruments 
In their data collection. Each tonl has Its strengths and weakness and 
each delivers a somewhat afferent kind and quality of daU. 

Finally* a suggestion that should be followed whenever possible pre- 
test the tools and instruments and data collection approacne* fn Pilot 
settings. Rehearsals are as important for the act of data celle* tlon as 
th*iy are for the acts of a, plav. (See Chanter X.) 

ft 
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Meljt work jhd related research plans 

hoposal for an evaluation study shouljhinclude plans for library 
research as we 1 1 as data collection from the field. 

If so*ie docune«ls> or reports will be needed, ilie evaluate should know 
where to find than, whu will have then, how to obtain copies of those 
Juuwjuks and now i»och jme* 11 Miylit requiie for obtaining those materials. 

Plans rot collection of field data should be made carefully, if the 
evaluator can not collect all the data personally, investigators or 
n interviewee fwy luye to be hired This means that plans must bb wade about 
their i ecrui luient and training. Local contacts In the field must be 
identified and orientation most be provided to them about the objectives 
of research and about research plans. 

Field visits most fit l(ie realities of the field and the convenience 
of individual respondents. Ihe evaluuor niist keep in mind Such considera- 
tions as the haivesting season, the weather* fairs and festivals and 
visits of V.I. P.' examination schedules in schools and training 
institutions, and planning and budgeting Cycles in departments and 
ministries. 

IV ob luns of traiiipoi tation should be all anticipated and solved. 
Keeping all of the preceding in view, a time schedule should be prepared. 
(t>ee Chapter XliU) 

L. i'lojris for da±a analyses 

Data analysis plans must alio form part of the proposal for tne 
evaluation study, will coding sheets or tabulations be needed for data 
* col ItM ton? K i>o. tt*c^e should be prepaied <md lasted. Personnel needed 
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for coding and collating data should be recruited and trained. Heed for 
technical consultancy or statistical help (even computer time, 1/ inquired) 
should be anticipated and plan* made for receiving such help. 

As In the case of data collection *ans* data analysis plans must also 
be Prepared In terms of a time schedule. Here Tfsts of things to be done 
Is not enough; plans must be tto^GnjJJjve. (see rhapters XIV and XV.) 

13. "Budgetary j>lans 

Conduct of an evaluation study will need statf time; secretarial and 
duplication help; paper, postage, taPe and taPe recorders (in some cases); 
field Investigators; and transportation and telephone costs* etc. 

ATI these resources exist with Jn training Institutions and programs 
and are available to those vho want to use them, tt Is impossible to 
think of a training Institution that would not want Us trainers to do the 
bes x training job Possible. Good training requires feedback; and* there- 
fore, evaluation Is an Integral part of all good training. The resources 
available in the Institution for "training" should be equally available 
tor the "evaluation of "training. " 

It can be said that the problem of resources K often self-created. 
Trainers may Took at evaluation of training as something separate from 
training and, therefore as an extra burden. When askn.) to do an evaluation 
of training they may ask for extra time. e*tra resources and extra credit 
for the evaluation work. 

The new oMentatlon towards evaluation as hart of trafnlng. will not, 
bowover, solve all resource problems* Institutions may not have budgeted 
for transport* Postage* stationary and secretarial help to include the 
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institution's ubl ijdt luns to evaluate its own programs. In such cases* 
extra budgetaiy provisiuns will have to be tuade and resources needed for 
iCoriduLLiity the evaluat ion study will have to be obtained. 

Ihu ptoposal for an I'vauation study should also include tile elettetH ur 
".^Purtiny plans/ Mill the eiaiuilion results he used within the program 
or tht I'lStitutiuii 01 will those h* disseminated outside the Institution? 
If di^cmi nation outside the Institution is envisaged, a clear description 
of uutside Lliencs and consumers of the evaluation study should be 
devolved. The same report 1j not necessarily appropriate for all groups* 
anil wi Uing dif foient versions of the report should be considered. 

In writing an evaluation report, the Policy and Program implications 
of daia should bo brought out. Data does not always speak for Itself, s 
VJltile it 1* nei_essa»y v that ^valuators bring out thP Implications of their 
findings \a\ policy makers and progrart planners* they should not draw 
uuwar i anted conclusions^ Op fans and hunches should not be infxtid with 
inferences ti^um the dati, 

Ivaluative m forma noil tarKbe buth used and abused. Too often 
nadci i of evaluation studies may be in search of culprits rather than 
Laus*.<*« arid may want tu punish rather hiin greaiei uinierjtuiluir.g 

in thw lulure* hu wouOei that col leagues' whose work Is being evaluated 
will often Jt-t wur« led about the evaluation process and what it might find, 
To handle Iht dopar bucntal politics of evaluation* It may be useful to 
dlstu^ the preliJuUtai/ report of evaluation in a group setting before v 
issuiiiij a linal evaluation report, 
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AH evaluation studies need not be duplicated and distributed, A 
single coPy of an evaluation study will be worth a thousand ■ if Its 
findings Illuminate action and If Us recommendations become part of 
decisionmaking', (See ChaPter XVI,) 

15, BjbUcgraphy * 

A proposal for an evaluation study should alsoMnclude a bibliography 
of books, reports and documents used In deve1opfngy\he proposal and likely 
to be used In the conduct of the study and in writing the' final report, 

Sujmary 

Ihe time and effort given to writing format proposals for evaluation 
studies Is often well spent, Elonents ia an evaluation Proposal may 
Include (0 the develojment context of (he training Program; (2) ttw 
description of the trafo\*jg Program In design terms; (J) the Problem 
complex; (4) the evaluation problem chosen for study; (5) justifications 
for choice; (6) review of available knowledge on the evaluation topic; 
(7) questions and sub-questions to be answered, (ft) the evaluation approach 
selected; (9) design* steps and procedures; (10) Instruments and tools; 
(11) plans for data collection; (12) plans for data analysts; (11) 
budgetary plans, (14) repOrti. 3 plans; and (15) bibliography ami H^t of 
resources, 

things to Do or Ihlnk About 

t, Prepare a formal proposal tor an evaluation Study of your choke* 
2- Haw* you condurted an evaluation study bef^ro? feo you think y^nr 
eval^,tlon stud/ could have oeen improved if a formal ptuposal had i>f»en 
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written before the diUtfl implementation of the ;t&Jy? If you have never 
conducted an evaluation study yourself, discuss the usefulness of the ideas 
included in this Chapter with someone who has. 



N 



CHAPTER VH 

\ * 
MOrUIORIflG AMD QUI CK APPRAISALS 

There, has been considerable Interest recently In development literature 
on strategies for gathering quick evaluative feedback on tbe perforate of 
programs. The point Is made that typical evaluation studies may too often 



Jta' e too long a time for decision makers to waft for results- Program 



At le-ast three different strategies can be suggested for such quick 
pulse-taking: 

(1) Monitoring* 

(2) Quick Apprafsalsi and 

(3) Networks or Informants. 

The 3*jcstiop we must ask here Is: Do mon1toiing t appraUing, and 
network building also requite written proposals much Hfce those suo^ested 
In the Previous chapter? The answer can not.be given In a simple 'Yes' or 
*Ho\ .For examptei a differs* sort of preparation wfll be needed^ for 
establishing a monitoring system which almost always w*J I be bi*1 it upon an 
adequate management Information syston (MIS). The dcve1o|jment.,of an MIS* 
In turn* will require extensive planning and development of profOrma\ for 
recording and reporting program implementation data. Quick appraisals* 
as wc will see tatpr, m%y require less exhaustive written proposals than 
the ones Suggested In ChapUr VI, bat will yet d^nand considerable formal 
Preparation. Planning for n^tworfcs of Informants will alsn rpqiilre fore- 
thnught ami preparatlnn 
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lo monitor is Lo watch or check upon an on-noing program Lo detect 
flaws diid to provide corrective actions. 

lluJkituMfiij (or ibiKurrviil MonUorlng) is typically based on data being 
ruutiiPtly ytmif dteil by <i program in tbe very process of Implementation at 
its various locations* levels and phases* 

Moitil-rUiy is done bust when assisted by a management Information 
sy^tuu (MIS) An MiS makes a program more monitorable and, thereby^ more 
i!!dn44»jable. 

liie following considerations must enter the design of an HIS: 
( 1 ) Qf jji 'lll aj >d untitl es where change wjjj appear * Such 

units >uay bu individuals (learners* trainerSi cocmmnUy leaders)* groups 
(women's clubs* discussion groups)* Institutions (health clinics, 
rehabilitation center*) * ur communities or sub-cul Lures . Sometimes these 
may bu physical entities* such as, Danes, fields* shops, wells, storage 
bins, etc, 

0 0 Cl|oji£_of null cdtors of chang e. Indicators or signs that 
*ii|iiit/ change in already selected units and entities must be chosen next. 
Jiit^e mdHatur* way attendance Ifl literacy classes, absence from the 
tattuiy fur i ua^ui.* ut I 11 -heal Lb, purchase of consumer goodsi rise In 
produtLlviLy, ^Ll, 

(ill) Uywity in ddtd Lul lect ion> In designing on HIS one need noL 
Luiic^t oil pu»»ivle JdLtf, lnd«d only the minimjfi necessary IfifofVMuiofl 
^huuld becfuju P<» L ul Lhu MIS. 

(m) jvltLtlyji i^Jints for J*rig!Ji£i!^i success ive aguroqat lun and 
UJ>jj*ju yt data, U ullouli* be clear where data will OrlginaLe* aL what 
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successive levels and poInU data will be aggregated and where it will be 
finally stored for possible retrieval and use In decision-making. This 
wMI require that data collection and data aggregation duties of various 
role performers in the program system are clearly Identified; and the 
logistics of the physical transfer of (fata are managed effectively. 

(v) Design of prof ormas, tabulatjo ns, report forms, etc. Simplicity 
and standardization of Instruments for data collection and reporting are 
Important for the successful implementation of an HIS. ' 

(vl) timing.. and flow 0 f ja ta « Monitoring falls If It Is not tlme- 
sensltlve. &ata should move In tlme^erle* according to a determined 
pattern of flow — upwards, downwards and horizontally as required, 

(vft) ttoU audit. Data audits are important to ensure that U^re Is 
no unreporting of -data and that the Intent of various proformas In use In 
the HIS Is well-understood. 

(vlil) Processing jind /eedback. Ihe data collected at a particular 
level should be used to Improve program oecfslons at that level before 
sending it upwards. When data is processed at the central level, 01 at 
one of the Intermediate levels, feedback should be provided to all concerned. 
Only then can the total system grow fr. intelligence. 

(2) Quick Appraisals 

Quick appraisals are quick evaluations* conducted under conditions of 
emergency to Investigate the cause of a b e*Mown t to anticipate problems, 
or to get early returns on the Impact of a program. 

Qnlrr appraisals may he less exhaustive and less comprehensive than 
regular evaluation studies* hut one needs to prepare for a g»ricfc appraisal 
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carefully and systematicdlly. A short form of the proposal as discussed 



in Chapter VI should be prepared. 

Appraisal teams will have to be carefully built, and given a clear 
mandate r&tftfidiuy, the information they shoti T J collect and the judgments 
they >!iy y ld render. Jhe team should think aoout the mix of the quantita- 
tive and qualitative data they will try to collect. 



The mstiuuents to he used for tfie quick appraisal should he carefully 
dusiijued 4iid pre- tested. Samples of respondents should be small but care- 
fully chosen. 

head-lines are important to i it the case of quick appraisals, 
otherwise* Uiey will not be quick appraisals any more, A time-frame of 4 
to 6 weeks is typical for quick appraisals. 

A quick appraisal is mure than monitoring which ases routinlzed 
data -jenurated in the very process of the implementot ion of a program. 
iiuicK jppidisalst vii the other hand, do collect fresh data for the purpose 
of aii-wciinj significant questions. However, whenever posslblet quick 
aPprai>als >huuld make secondary use of datd already stoied In the manage- 
ment information systtau. 

^elf re^orts by fomtioiiar les Of a program on their own performance 



i> diutner important icsourie.for quick appraisal teams. Slmildrlyt selfV 
rej^oi t> by the belief 1 1 1 ar les of a program should be used for developing 
Mulct appraisals. 

(J) '"ddptinjl Mass Observation 
tk> Networks of Informants 

i'iurc»uf rtjihje of Gitat Britain developed an interesting tWdback 
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technique that he Iras called "*Mass Observation," The mass ubservatwn 
technique uses untrained volunteer reporters who make their observations 
In the normal coarse of thefr lives and In the natural course of programs 
and events. Then they describe in trrlting, fully* clearly and simply what 
they have ohsprved ami what they think their observation add up to* Such 
reports may Include photographs* clippings fron the focal newspaper* 
things heard over the local radio station, or overheard In the local bar, , " 
Sometimes trained Interviewers may also be added to the corps of people 
observers* Advocates of thfs technique believe that It Is easier to detect 
the bias of an untrained observer than that of a social scientist and 
suggest that mass observation should not be dismissed as a biased approach 
to the collection of feedback. They assert that it Is an excellent approach 
to the study of the "mind life" of the people. 

The mass observation approach can be adapted to developing sbeiial 
netwoiks of Informers *ho do a limited version of the mass observation for 
the ^valuators, A whole group of people located at various places and 
levels of a program can be asked to observe and write about a program, 
■ Its process* its Impact and consequences, 

Swnary 

Urgency of the need foi* fnedbacfc m^ demaml quick pulse-taking rather 
than systwnatlc evaluation of the Program, Three different approaches can 
be use*! fqr qoneratlng Sur,h feedback; (1) monitoring, {?) quick aPbralsats, 
and (3) networks of informants. Each of these approaches will retire 
adequate preparation, even thouqh formal proposal writing as suggested fn 
ChaPtec VI may not be undertaken, 

8J 
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Jhjjigs to Ito or llitnk j jjout 

1, Cjm your training center or institute make use of an HIS? What 
wight soue of the essential elements of such an HIS? * 
l> lis>l the n^ies, official designations or occupations of at least ten, r 
L>u?pk you would include in your network uf Informants to collect "feedback 
on yom piugiam. What special contribution yjou expect each of than to make? 
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CHAPJER VIII 

MODELS Of EVALUATION 1 

i 

In the literature on evaluation, one is likely to come across the 
terms 'evaluation approaches', 'evaluation models' and evaluation 
paradigms'. An undei standing of the meanings of these terms Is essential 
Tor a thorough comprehension of the ideas jpyesented \t\ this chapter. 

Let us begin with the word 'paradigm'. In the dictionary meanings 
of the word, a paradigm Is an ordered Hst, a table of classes, a fratterri, 
or a formula for the general form into which specifics of a certain order 
may he placed. 

Kuhn 1 In his stuoV of scientific revolutions gave a special meaning 
to the word paradigm. He defined a paradigm as the "creative ideology" of 
scientists from which they worked, an (J which provided them with a 
particular logical and methodological stance for Producing scientific or 
social scientific knowledge, ^ 

Thus* evaluation paradigms are the crea* ideologies of eyalu^ors. 
These paradigms determine the thinking and methodological behaviors of 

£ tn t Udi ui > < filial Liicy thlli* atlut the rtatiifc Of Ttv\\Xj\ <Sl*d hCW thCj 

think dc ?ndable statements can he made about stability and change In the 
social reality* that surrounds rfs* There are two basic paradigms of 
evaluation (and research) that we will be discussing later, the scientific 
parad'gm and the naturalistic paradigm* 

'f, Kuhn, The Stru cture of Scientific Revolutions. Chicago: 
University of Chicago Pressi 1552'. 
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- tft: should define moduli next. A model is information, data or 
principles yrouped, vei bally or graphically (and sonnet Sines mathematically) 
to represent or describe a certain thing, idea, condition or phenomena. 
A model is the essence of the learning and thinking of a specialist, 
stated clearly and pars imoniously. Models are the progeny of paradigms. 
It is i^p^rtant to*emember this relationship between paradigms and models. 

Evaluation iwdels are thus verba), tabular or graphic presentations 
of the pnuciples learned by evaluators. They are the essence of their 
Separate experiences developed in the background of particolar paradigms. 

Finally, ao approach is merely a method of beginning or accomplishing 
suiMethmy. Muwever^when an approach, through successive ose and testing, 
becomes both standardized and formalized. It acquires the status of a 
rtodelJ 

Two tigs ic Pafddijiujs of Eval uatio n 
E dyygUQh a nd Dey elopmejU 
Tiiers ate two basic paradigms of evaluation in education and develop- 
ment: 

1 the Scientific Paradigm of (.valuation, and 
2. TJie flatorahsuc Paradigm of Evaluation. 



In our discossion ut this chapter, the term 'theory* has not been 
defined ur explained. This is so because the litei^ture on evaluation 
Ulks often of evaluation Models and seldom of evaluation theory. 'Let 
us say briefly that, in terms or a conceptual status, theory falls betwe&i 
the paradigm and the model, in its best sense, theory is a deductively 
fruinectei* *et of laws and empirical generalisations. A model is often a 
schematic diagram that connects theory with practice, 

9o 
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The scientific paradigm 

The scientific paradigm Is so called because It copies the methods of 
hard sciences: physics, chemistry* botany, engineering, etc* 

It Is sometimes called the classical paradigm because It Is oldt 
follows strict rules and Is seen as standard and authoritative. 

It has also been referred to as the positivism paradigm because of 
Its emphasis on the explicit; that which Is capable of being directly and 
certainly affirmed* 

The 4 Ideal °f the scientific paradigm Is the randomized controlled 
experiment. (Quasi-experimental designs may be acceptable under some 
conditions.) The Scientific paradigm demands a clear definition of evalua 
tion objectives* of variables* a sampling plant structured instrumentation 
that generates quantitative data* statistical techniques In the analysis 
♦ of data and generallzablllty of results. 

T he naturalistic paradigm 

<* 

The naturalistic paradigm Suggests that human bahavlor be studied In 
natural settings* within Its total context. It Is the method of the 
anthropologist and the ethnographer, it Is holistic - It studies reality 
whpiet wl thout dividing It artificially Into Parts and segments. 

The naturalistic paradigm Is a1£o referred to as qualitative and 
phenomenologfcaT. This means that* unlike the positfvlst* the naturallstl 
evaluator seeks to describe phenomena and search for regularities and 
patterns, lie searches for understandings of the specific situation that 
may later Illuminate other soirewltat similar situations, tie does not 
search for generallzable laws, " ^ 
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In naturalistic inquiry, the design i; emergent; It emerges as the 
evaluator undertakes different steps and procedures* In the collection of r 
meaningful data. The samples are purposeful rather than r random, The 
Instruments are always unstructured and generate qualitative data* Claims 
4 are ode in regard to the applicability and fittlngness of res»ks rather 
than to their generallzabillty, 

9 

The scientific Par^fgm versus ( 
the naturalistic paradigm 

t The scientific paradigm has had great victories In the hard sciences, 
Jt has produced research that has banished diseases from the face of the 
earth and has put man on the moon, 

Jt was so successful that social scientists (sociologists* psychol- 
ogists* economists* educators* even anthropologist s)* also wanted to use 
the scientific paradigm of the physicist and botanist. They used tt with 
a vengence, Jt made them feel like real scientists, for years and years, 
the scientific method wes learned and the scientific method was taught In 
all the social science departments of unJ/ersitles. 

The realization has emerged during the last twenty years or so that 
the scientific paradigm has given social scientists good feelings but not 
good scientific results, We have discovered Ahat too often social Hfe 
does not fit Into the laboratory. In trying to control variables* we 
segment human behavior unnaturally and Indeed change the very nature of the 
human behavior being studied. Aggregation of scores and statistical 
treatments of dati may look elegent and Impressive but results may be 
trivial and even misleading. In human behavior the context i5L important. 
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We need to study not Just behavior but behavlor-ln-context. 
* The naturalistic paradigm Is more appropriate most of the time, for 
the study of human behavior. Once rejected out of hand as subjective 
and qualitative it Is becoming more and more acceptable. As Its - * - 
methodology becomes clearer and techniques of data analysis are further 
advancedt the naturalistic paradigm of evaluation will find Us rightful 
plaa In evaluation methodology. 

The readers of this book should not select one of the two paradigms 
discussed above and reject the ottier* They should learn to consider them 
to be complementary. The scientific paradigm is necessary when developing 
profiles of large-scale programs for use by decision makers at the center. 
The naturalistic paradigm Is needed when the emphasis Is on an understanding 
of processes and effects at program levels. Typically, educational 
evaluators will be using both these paradigms within the context of the 
same one evaluation study, 

The following table sumnarlzes the differences between the two 
paradigms of evaluation: 

TABLE 1. DIFFERENCES BETWEEN THE TWO BASIC PARAOIGMS OF EVALUATION 
The Scientific Paradigm The Naturalistic Paradigm 

Philosophic roots 

Posltlvlstt phenomeno logical, 

reductionist holistic 



Experimental or 
quasi -experimental 



Emergent design (or 
rol ling design) 




9B 



Sett1n9_of evaluation/research 



/ Laboratory or 

' otherwise controlled 

Sampling ' 

RanoW She 
pre- determined 



Methodological orientation 

Quantitative 

Instrumentation 

Structured* often 
.Interventionist. Instruments 
are sought to be standardized 
arid nude independent of 
e valuator 

Data analysis ■ 

Statistical 



Report 
Statistical-analytical 

Strengths 

Provides good estimates of 
differences* variations* 1 
and correlations when ' 
variables can indeed be 
properly defined and reasonable 
controls can ;tabllshed 



Ecological; In context 



Purposttive* elite* 
specialized. Size determined In 
use; sample Is exhausted when avail* 
able Information Is exhausted 



Qualitative^ thick description 



Unstructured, often unobstruslve. 
ftesearcher/evaluator himself or 
herself becones the tool of data 
col lection 



Thematic; content analysis of 
interviews* documents* and 
observations 



Descriptive* Interpretive; 
typically a case study 



Responsive* adaptability, holistic 
emphasis 



Guba and Lincoln In their recent book** have discussed the various 



'tgon G, Guba and Yvonna S, Lincoln* E ffective Evaluation* Improvin g 
the Usefulness of Evaluation ties u Us ThroyjlT^b spans lye and tiaturalistlc 
Approaches, San Francisco* Ca*; Jossey-Bass» ml. {page 104) 
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aspects of rigor as they aPPly to the two basic paradigms of evaluation. 
What they are saying Is that both the scientific and the naturalistic 
paradigms of evaluation can bo rigorous In their own ways. See tabic 
below. 

TABLE 2, SCieNTIFlC AND NATURALISTIC TERMS APPROPRIATE TO VARIOUS 
ASPECTS OF RIGOR ^ , 

Aspect Scientific Term Naturalistic Term 

Truth value - 
Applicability 

Consistency 
Neutrality 



Hode1s_of Evaluation 

Against the background of these two general paradigms, many different 
models of evaluation have been proposed by specialists In the field. 

Why are there so many evaluation models? Isn't there one correct 
way of doing evaluation? 

Earlier In this chapter * we defined a model as the essence of the 
learning and thinking of a specialist* stated clearly and parsimoniously 
for comwnl cation among professionals and practitioners. There are many 
different evaluation models* because different specialists have undergone 
somewhat different learning and thinking. Ihey have accepted one basic 
evaluation paradigm or the uther* Bore and more evaluation specialists 
now suggest a pragmatic mix of the two paradigms - the scientific and the 
naturalistic. 

101 



Internal validity Credibility 

External validity/ Flttlftbness 
generallzabllfty 

Reliability Audltabltfty * 

Object y\ ty Conf 1 rmabf 1 1 ty 
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- Evaluation models are different also because they have emerged within 
different program settings: within formal education or within out-of- 



school and nonfoiml education settings; within mental health setting in 
an Industrialized country or wkhfn family life education In the context 
of a developing country. ^ 

finally, and most importantly, evaluation models are different 
because evaluation specialist have Introduced additional "value" consldera- 
tlons to their Initial choices of paradigms. Soue evaluation models 
emphasize a more synoptic view of evaluation, suggesting that we evaluate 
not only the behavloV of our so-called clients but also of our own. Some 
evaluation models suggest, the Introduction of Imagination* to our evaluations 
and not depend only on cold calculation, lorn suggest that the unanticipated 
consequences of program actions may be as Important as the intended and the 
anticipated. Therefore* the model of evaluation should be able to o 
% accommodate both the anticipated and the unanticipated consequence. Some 
Suggest evaluation to be conducted as an advocacy and confrontation. Some 
suggest participative evaluation wherein both the means and ends of 
evaluation are particlpatlvely determined by all concerned - organizers' 
/ professionals, and beneficiaries. ^ 

One can see a clear underlying value direction In the development of 
evaluation models during the last twenty years; (4) there Is exclusive or 
complementary use of naturalistic strategies; and (2) there Is a move 
^towards Inclusion of the beneficiaries of programs In the design and 
Implementation of evaluations. The key words are holistic and participative. 

Some of the evaluation models often referred to In the literature of 
evaluation will be discussed below. The discussions will be brief. We 





; 101 

Include a discussion of the evaluation models for two reasons: educational 
and political, the tralner-evaluator should have Some 'idea of what 
different evaluation models exist and what their characteristics are. litis 
Is for his or her education. But there Is also a political reason. The 
tralner-evaluator should be able to justify his or her choices of the 
model or models; and should be able to hold his or her own against t>»e 
outside specialist. He should not allow technicians and specialists to 

browbeat us with the use of unfamiliar names and phrases. 

•* 

Before we proceed with Short descriptions of.evaluatlon models* we 

^ * 

should caution readers against a search for the model, for one correct way 
of doing evaluation. As fn the case of paradigms, a pragmatic mixture of 
models may be often the tast thing to try. 

The Context- lnMtjroces s --Product {CjPPlHodel of Evaluation 

This model Is often associated with the name of Daniel L. Sttifflebeam 
who has used this model In various evaluation studies. 

4 

According to this model* the sole purpose of evaluation Is to produce 
Information for decision makers. Using the system metaphor and the four 
parameters of systems (context* Input* process and output)* the model 
talks of four types of evaluations to provide Information for four types 
of decisions: 

1. Context evaluation - to provide Information on the setting, to 

be able to make planning decisions; 
Z. Input evaluation - to make programming decisions; 
3. Process evaluation - to make decisions related to methodologies 

and Implementation; and 
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4, Product evaluation - to evaluate tinpact and to make recycling 
decision. 

The ClfP picwlel corner cFosest to formal research models with its stress 
* on the clarification of evaluation objectives* structured observation* and 
the testing tradition of achievement testing in schools, The model adopts 
the criteria of Internal and external validity* reliability, objectivity* 
relevance* iapurtance* scope, credibility, timeliness* pervasiveness and 
efficiency of the evaluative information produced. It has* howeveri shown 
little concern for values, The *odel suggests that evaluators should ' 4 
produce reports that decision makers can use in making decisions, 

« 

The Countenance Model of Evaluation 

The countenance ittdel of evaluation is associated with the name of 
Robert E, Stake, It is so called because Stake talked of two countenances 
of evaluation - description and judgment. 

This model was directly related to the evaluation of effects In terns 
of stated objectives and Involves the completion of two data matrices as"* 
follows; 



Description Judgment 
Hatrix\ Matrix 





Intents 


Observations 


'Standards 


Judgments 


Antecedents 




* 






Transactions 










Outcomes 
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The task of the evaluator is to find data for all the cells In the * 
table on the previous page: £o compare observations to Intents; and to make, 
judgments In terms- pf the standards agree&to among program organizers and 
evaTuators, One should note \h at In systems vocabulary antecedents are 
Inputs and transactions are processes, 

Themodel In Implementation has used stratified random samples for 
collecting special Information^ combined with the case stuoV approach. 

The model has called attention of ^valuators to the need for defining 
standards on tbe basis of which Judo^ents^can be^made, though the model 
Itself has left the Question of specification^ standards unresolved. 

The Olscrepancy EvaluatlogJtode l 

The model *&s proposed by Malcolm Provus who defined evaluation as the 
art of describing a discrepancy between expectation and performance of a 
program. 

+ \ 

The model suggests that we look for discrepancies In terms of ffve 

different aspects of a program: 

* 1, the design of the program 

2, Its Installation 

3. the processes of Implementation 
4# tTie product* anr* 

5* the cost *■ 

While recogplzlng the usefulness of the experimental method In 
certain casesi the model shows preference for the descriptive methods of 
history and anthropology and the case study method of sociology and 
psychiatry. Thus* the model accomodates both the, scientific and the 
naturalistic paradigms, 1 > 

9 106' 
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With Us relative emphasis on naturalistic methods* it suggests that 

^ • • 
evaluaturs work in tews to be able to test individual perceptions of 

each against the other and to be able to question the standard* being 

applied. to describe discrepancies. , / 

The model claims to provide continuous information tofdeclsion makers 

on the performance of an on-going progran. It also claims to provide 

^information, that has direct one-to-one relationship to decisions actually 

. being made. The resources required for effective application of this 

model can bi 1 considerable in terms of personnel* tfme^nd money. 

The Transactional Evaluation Model 

The transactional evaluation model is associated with the name uf 4 
Robert K ; Rippey who has challenged educators and trainers to show what 
they have accomplished rather than what scores their students and trainees 
■ have made. * 

The focus is on educational accountability - change makers are 
osked to stuoy thefuselves» their roles» the system in which they play 
these roles and the larger systems that surround t the systems under change. 

In Rippey*s 1 own words; 

i* * 

A comparison with traditional summatfve and formative evaluations 
show that the target of evaluation Is different; the subject of evaluation 
is the system, not the client or the services rendered by the system. The 
variables relate to the social > psychological and communication aspects of 
the system* rather than to the manifest objectives. The information is 
continuously fed back into the system. The tvajuator himself is more a 
part of the operating system. The conventional considerations of 
reliability* validity and objectivity are less important than those of 
timeliness, relevance and the observable effects of generating evaluation 
information, w Primarily* evaluation Is Intended to transform the conflict 
^, r — 

1 Robert H. Rippey, ed.» Studies in Transactional Evaluation , 
Berkeley* Ca,; McCutchan, 1973> pages 3-4, 
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Energy of change Into product! ve.actlvlty; to clarify the roles of those 
" persons Involved In the program changes , not to produce new knowledge or - 
ascribe causality* / * 

One should note the assumptions In regard to the basJc paracftgm.m 

-use In the transactions] evaluation model and the additional value 

positions Introduced \n the model. It Is Indeed a highly value-laden 

model. It emphasise; relational Information and urges sensitivity to the 

unanticipated consequences. It also implies that evaluation be conducted. 

collectively by the protagonists and designers of a change program and 

representatives of those likely to be affected, * ■ * * 

The Goal-Free Evaluation Model ■ 

The Idea of goal-free evaluation was introduced by Mfcheel Scrlvon. 

He pointed out that In our emphasis on stated goals* our search got 

completely focussed on Intended effects - effects we wanted to create 

under accepted program 3ga.ls. This focus became $o exclusive that we 

often developed a tunnel vision: looking for evidence of Intended effects 

* * . * 

and seeing nothing else. , * 

He Suggested that we should Took for the real effects of programs - 
effects that had actually occurred whether intended or unintended. This he^ 
thought could be done if we conceived of a goal-free evaluation, 
independent of objectives stated for programs. Results Vrom objectives- 
focused evaluation and goal-free evaluation of a program could then he 
combined... ■ 

The investigative Social Research Model 

Jack D. Douglas 1 has recently analyzed the methods* of the Investigator 

'* — ' zr 

back D. Douglas, Investigative 0 Social Research . Beverly Hills, Ca.i 
Sage Publications, 1976* 



ERJC ' * 

407 



106 

and show how Investigative strategies can be used to expose the truth about 

O 

people in*social settings. x , * 

- . ** ^*ito- 

The Investigative model does not assume a world of cooperation, open- - 

ness and truthfulness/ hut one of misinformation! evaslonsi lies and fronts, 
Ke 'then' suggests strategies for grasping an evaluation setting* Infiltrating 
tne settlngi building friendly and trusting relationships) and then using 
them in a continuous process of testing out and checking out, * 

The modus operandi *aode) rsyggestedifctf Michael Scriven is also *n " 
Investigative method for studying causceffect relationships, through 
sequential testing. This method reconstructs "the procedures of the 
h1storian> the detective, the anthropologist* and the engineering trouble- 
shooter/ The modus operandi model fs proposed as a substitute for 

experimental and quasl-experl mental Approaches when field situations 

\ 1 & 
preclude their use. Essentially, the method involves generating 

hypothetical chains of cause-effect events and eliminating those that 

could. not'possibly have happened. This is the topical net hod of the 

detective-"' 

" Evaluation as njuttjnajjon* 

This model was developed in clear rejection of the "agricultural- 
botany* 1 model of evaluation rooted In the scientific paradigm, it was, 
asserted that groups and comuwftities »n not be randomly assigned to 
treatments as farms and*fields; and human beings can not &e administered 
treatments as seeds in the ground* in any case* quantitative data genera Ud 

-by. the agrlcultunfl-botany model* provided-. only partial descriptions of 

, phenomena, " * 
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Parlett and Hamilton 1 built their model on two Important considera- 
tions. 

1. Instructional systems t once adopted^ become living systems*, living 
systems do not match their catalog descriptions* Important modifications 
occur In programs as they move from the drawing board to actual 
Implementation* 

2. Programs of training and development can not |>e separated from their 
■learning milieu. Actors In the learning milieu and\he structures of the 
Milieu become part of the Instructional system. \ 

Hhlle retaining the use of sampling methods* and structured 
questionnaires and tests, Parlett and Hamilton drew our attention to the > 
naturalistic methods for description and interpretation. Ihree stages In 
the evaluation process are suggested to Include: (a) observation of the 
educational setting; (b) selection of themes through progressive focusing 
and Intensive Inquiry; * n d (c) analysis' and explanation. 

The Connoisseur ship Model of Evaluation * - 

Ihe connolsseui^p ribdel of evaluation proposed by Elliot W. Eisner 2 
makes a clean break with -the scientific paradigm and draws .from the 
aesthetic tradition of the arts. 

Eisner suggests two Interrelated concepts: (a) educational 
connoisseur-ship and (2) educational criticism to perform the tasks of 

IHiJEazleJtt andJ)*„H*mi-Uon^ "Evaluation as Illumination: A New — - 

Approach to the Study of Innovatory Programs* 11 Occasional Paper* Ho. 9* 
Edinburgh; Center for Research In the Educational Sciences* University 
of Edinburgh* 1972* 

*Elt1ot H. Eisner, Educational imagination: Ihe Design and Evaluation 
of School Programs . Hew York: Macmlllan Publishing Co** 1979* , 
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educational evaluation. Educatlonalcorutoisseurshlp is the ''means through 
which the shape of the context and the configurations within H can be 
reorganized so that intelligent decisions about the context can be made,* 
EducatiPnal criticism is the art of disclosure through description, 
interpretation and evaluation, * - 1 

The methodology of connoisseurship and criticism is by no means soft 4 
. headed or romantic, and certainly can be systematic and rigorous. Educa- 
tional critics can learn to look for the pervasive qualities of education 
in the classrooms and training settings; and can learn to look for the 
meanings of bidden cues* 

Questions of reliability and validity must behandled through 
structural corroboration (mutual validation of bits of data one by the , 
rest; and the whole being supported by the bits that constitute it); and 
referential adequacy (the existence of relationship between what the 
educational critic says and the subject matter of his or her critique). 
Generalizations are also possible In the sense that educational criticism 
will lead to more refined processes of perception in subsequent settings* 
and will create i n the evaluator's mind new anticipations. 

Reports of educational criticisms have a family resentence to case 
studies, but case studies of educational criticism are different In the 
sense that criticism itself Is an art form. As a critical disclosure, 
educationa-1 criticism report creates a living image* communicating to Its 
readers a visceral understanding of "Uie" educational realities. — 

Tlie Advocacy Model of Evaluation 

The advocacy mode) is given this name since It allows groups of 
people both for and against a program to advocate, their position before 

erJc : . HQ 
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an educational jury. This model Is, thus* evaluation In the legal mode. 
It Is an educational trial, by jury. 

An educational jury Is established and groups with opposing views on 
the usefulness of a program argue their cdse before the jury. All kinds of 
data quantitative and qualitative can be presented and any witnesses may be 
called to testify* These witnesses may Include specialists *nct non* 
specialists* organizers and beneficiaries* tten and women* and adults and 
children. 

The model seeks to bring out both sides of an Issue In a balanced 

L 1 

fashion* but wost cf Us advantages are counterbalanced by corresponding 
disadvantages. The model Introduces controversy and confrontation where 
none may have existed. Good juries are hard to put together* The format 
does* however* force administrators to consider that there may be some 
things wrong with the 0n-go1ng*program and to consider the possibility of 
discontinuation or drastic transforation of a program. An Important 
advantage of the model Is that evaluative Information Is used to come 
to concrete decisions In .regard to the continuation or re-design of 
programs. 

The Participative Evaluation Model 

The name of Paulo Frelre* the Brazilian educator and the author of 

r 

Pedagogy of the Qpressed (Herder and Herder* Hew York, WZ) Is often 
associated with participatory evaluation and research. Considerable 
amount of work has since been done In this area during the last ten-years — 
by evaluators spread all over the world* Participatory research networks 

Vartlctfialoxy Research: Developments and Issues . A special Issue ' 
of Convergence * Vol. XIV* Ho* 3* 198K 
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have been established, participatory research and evaluation studies have 
been conducted and their resuUs published. 

Participatory research or evaluation Is not a scientific endeavor of 

v 

the professionals* but an in-depth, existential review of an experience 
done by all concerned* together In mutual collaboration. The learner 
becomes an e valuator and the evaluator becomes a learner. Evaluation 
goals t ends, standards and tools are decided upon partlcipatively. Each 
contributes personal data and collects the data that has to be obtained. 
Analysis of data is collectively undertaken. Judgments Are also rendered 
collectively- 

ln*an address to the institute of Adult Education* University of 
Qar-es-Sa1aam» on £0 July 1972* Paulo Frelre presented the possible steps 
In such a participative methodology; * 

1, Tne evaluation (or research) team should acquaint itself with all 
previous research and evaluation - no matter what methods were used In 
that research evaluation. 

2, The team should delimit the area of action geographically - even 
though* culturally speaking* there are no frontiers, 

3, The team should identify official and popular institutions In th§ 
area selected and go to talk to the leaders within those institutions. 
4* The evaluation teaw should tell these leaders* in all honesty, that 
they have corue to discuss the possibility of aU people In that community 
holding discussions and working together. 

5, If the leaders agree* the evaluation team should hold meetings not 
only with the leaders of various Institutions but also with thf> people 
who are involved In some way with those Institutions, 



112 



0 



111 

6. The evaluation team should discuss with th£ community arrangements 
for meetings wherein groups of* say* thirty people could come together on 
a dally or weekly basis for discussions. Such meetings might Involve 
almost all the Inhabitants of a community and last for several weeks. The 
Important thing would be to obtain a perception of the whole community* 

7, Sociologists! psychologists, educators and linguists should* at this 
stage* Join the research or evaluation team and visit each group. Records 
of discussions should be made at each meeting* People shoi'ld be urged to 
speak If they are silent* but otherwjse the role of the evaluation team 
should be no more than advisor/. One of the members of the community 
should chair such meetings, 

8. Justice* education* government* Industry and many other topics may 

be discussed* but all In terms of the people and-ltt the context of concrete 
realities, 

9, When the smaller groups think they have exhausted the topics for 
discussion* each one should put its findings on paper and then they 
should all meet In a general session. The reporters at such sessions 
should be the people themselves; not the specialists on the team. The 
workers should become Intellectuals, There should be collective discussion 
of each group report. * 

TO, The evaluation team should now make a critical study of the 
people's discourse* This study should be Interdisciplinary, The various 
levels at which people perceive reality must be determined and their 
many Indications should be worked out. These Implications must be 
studied In the presence of the people* not by social scientists on their 
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11. Tito evaluation team together with the people should now draft a 
proposal for subsequent^ action. The program* itself should not be f 
worked out for the people but with the people, 

H should be clear from the preceding that participative evaluation 
Is not distinguishable from need assessment or couountty awareness. The 
distinction between evaluation and instruction gets lost In participative 
evaluation. Participative evaluation provides participants with further 
opportunities to raise their consciousness and consolidate their sense of 
power and se1F»worth, 

The Situation-Specific Strategy (3-S> 
Hodel of Evaluation 

Before presenting the 3-S evaluation model* let us remind our\readers 
that lt ls useless toloGK for the model of evaluatlon^or for one -correct 
vay of evaluating training or development, A trainer-evaluator mlojit 
often, be using more than one of the above models, within the context of 
a single evaluation study, \^ 

Another Important point to remember Is that models are not usable as 
formulas, All of your problems and needs will seldom fit neatly into an 
evaluation model , Models should Indeed be used to "think with", not as 
step by step and unchangeable sets of procedures. 

The 3-S model to be discussed below Is a mode) that should help us 
select a good mix of models and approaches to be used In an evaluation 
study. The conceptual essence of the 3-S model Is this: Do not start 
with the evaluation model* begin with the evaluation problem. Analyze 
the evaluation problem Into stitHproblems* think how the problem or parts 
of the problem ml9ht unfoW over time; and* finally* think of the milieu 
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In which evaluation will be conducted. 

Different parts of the evaluation problem will most likely require 
different evaluation models and approaches. YoM may need both a survey 
and an In-depth case study. You tray require achievement testing as well 
as content. analysts of documents. -> 

The exigencies of time may demand pulse- taking throughout ck appraisal 
even though. Ideally, a more systematic evaluation would have been better. 
Finally, the evaluator may be working In a situation where there are no 
calculators or colleagues who can help with the analysts of large bodies, 
of numerical data; where there are no copying machines or stencil 
duplicators; or where there ts no duplicating paper for producing the 
required Instruments. The 3-S model helps us think about what strategies 
to choose In specific real-life situations. How to do the second best 
when the best I s no t possibl e? 

Elsewhere.^ we have listed the following steps In the Implementation 
pf the 3-S evaluation model: , * > 

1. Ordering the world of change and evaluation (Chapters II and HO 

2. Articulating the means-ends relationships In the change program,. 
(Chapter IV) 

3. Generating profiles of Information needs and evaluation issues 
(Chapter V) 

4* Developing a situation-specific evaluation agenda (Chapter V), find 
5. Choosing appropriate and realistic methodologies and techniques 
{Chapter VII I and subsequent chapters). 

V S, Bhola, Evaluating Functional Literacy . Amersham, Bucks, 
tU.: Hulton Educational Publications Ltd., I?"?* Pages 25-33* 
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There are two basic paradigms of research and evaluation; (1) the 
scientific oaradigm and (2) the naturalistic paradigm. Against, the back- 
ground of these two basic paradigms* a variety of evaluation models have 
been proposed* among them; the C1PP model* the countenance model, the 
discrepancy model* the transactional model, the goat-free model* the 
Investigative model and the modus operandi approach* evaluation as 
illumination*^ the connoisseurship model* the advocacy model* the 
participative model and the situation -specific strategy (3-S) model. The 
trainer- e valuator has to understand the strengths and weaknesses of the 
various paradigms* models and approaches and learn to develop conceptual^ 
strategies that fit his or her problems in the particular context. 

Things to Do or Ijrlnk About 

Sic paradigms discussed In this chapter, which is likely 



Jto generate more useful information on your training program? Or» do you 
have to use both? 

2, Of the models described_1n this chapter, which model or models do you 
personally consider most useful In your work? What more would you like to 
know about the model to put it to work? ^ 

3, Can you find evaluation studies already completed that fit neatly 
under one or the other model described in this chapter? 
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CHAPTER IX 

CONCEPT ANALYSIS* INOlfcATORS OF CHANGE 
AND STAWJAROS OF.QUAUTY 

In an earlier chapter t we talked of raising evaluation questtonst of 
developing evaluation agendas and about typical evaluation themes of Interest 
to tralners-evatuators. tet us examine here* some of the evaluation Ques- 
tions that tralners-evaltiators have actually asked themselves and the evalu- 
ation topics that they have set out to study: 1 

t. To determine the relative effectiveness of different sets of post- 
literacy teaching-learning materials being used In the programs for new 
literates * In terms of content, occupational orientation* style and format . 
Z* An evaluation of the efficiency and effectiveness of the "Administrative 
Skills Courses for Chiefs* offered by the Government Training Institute 
(GTjh Haseno. Kenya* during 1973-77. 

3* An evaluation of the basic education curricula for the Undugu schools 
In Nairobi, 

4, An evaluation of the resettlement programs for the vocationally-trained 
disabled persons In terms of their social and economic rehabilitation . 

5, An evaluation of the effectiveness of the Form W literature- In-Engtlsh 
radio program In the Central Province of Kenya. * 

6, An evaluation of the suitability , and the utilization of available fceal^j 
education materials by the Family Health Educators In Nafcuru; and an explora- 
tion of the need for producing materials suitable for local use, 

*These examples have been taken from the evaluation questions and topics 
selected by the participants of Hay 1979 and August 1980 workshops on the 
Evaluation of Oaslc Education and Development Training Programs* Ker1cfao t 
Kenya. 
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7. Reasons for dropouts from the diploma education training program at the 
National Institute of Education* Hakerere* Uganda. 

8. To evaluate the Puterns and frequency of Hbrfrrr use by students at 

Hachakos Teachers' College* Kenya. 
* ■ r 

9. Evaluation of the effectiveness oftfohe Nutrition Surveillance Curriculum 
for the July 1980~March 1961 class In the Nutrition Course at Karen Col1e9e* 
Kenya. J ^ 

10. (tow effective is the utilization of handicraft skills gWfen to field 
Workers in improving the nutrition status of the perl-urban communities that 
these field workers serve In the Nairobi area. 

The preceding examples'provlde us with a snail frample of the types of 
evaluation questions that can be asked by tplners-evaluators. But In this 
chapter* we are Interested In the words and pjjrases used In these various 

statements of evaluation topics* Let us examine the words and phrases which 

i 

have been underlined In the examples above; * 

6 basic eduratlpn 



determine 
(example* 1) 

relative 
(example* 1) 

effectiveness 
(examples* 1 b 2 b 
5, 9) 

new literates 
(example* ]) 

content 
(example* 1) 

format 
(example* 1) 

efficiency 
(example* 2} 



asic eduratton 
(example* 3) 

curricula 
(example* 3) 

disabled person 
(example* f) 

social and economic 
rehabilitation 
(example, 4) 4 f 

suitability 
(example* 6) 

health education \ 
materials 
(exapiple* 6) 

dropouts 
(example* 7) 



library use 
(example* 8) 

nutrition 
surveillance 
(example* 9) 

handicraft skills 
(example* 10) 

nutrition status 
(example* 10) 
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Almost all of the words and phrases Ms ted" above can he understood by 
* » 

an educated person who knows the English language* An e valuator* however > 
can not deal with all words In their common meanings • For Instance* In 
example 1 above* words "determine* 1 and "relative" may be used in tne dictionary 
meanings of the words* but no£ such words as "effectiveness", "new literates"; 
"content" or "format". These words will have to be specially defined by the" 
tra1ner~evaluatOr within the program context of Ms or her study. , 

lekus look at some of the other words and phrases Jlsted above. What 
do we mean by "effectiveness"? Wouldn't "effectiveness 11 mean different things 
«, in each of the four examples (ly2 t 5 and g 'above); and be Indicated the 
appearance of different kinds of behaviors? It certainly would. What Is 
a "new literate 1 *? One who comes out of an aduTt literacy class or anyone f who 
acquired his or her literacy recently? What level of literacy 'will , be 



a cep table for a person to be called "new literate"? What "content" will be 
considered In the evaluation study - only the "content" of primers or of all 
reading materials? What aspects of "format" will be Included In the study 0 
Of Instructional materials? What qualities will be examined In making 
Judgments about. '•format"? * 

What Is a "disabled person**? In the particular study referred to above 
only cripples were considered disabled. What 3s "social and economic 
rehabll Itatldn"? Should the cripple be able to 0et married* become a village 
leader* and have his own house or hut to be considered "socially and 
economically rehabilitated"? What Is a "dropout"? Is a "dropout" different 
ofrom a "pushout" - the student who was asked to leave because he or she 
failed on the Intermediate examination? * 



113 

ERIC 



118 

■ ' ' t 

What constitutes "library use"?. Would the borrowing of a prescribed 
textbdok from the library constitute acceptable "library use"? Uhat is ^ 
"nutrition surveillance"? How is it different from "nutrition*' status"? 
What skills wMI be included in "handicraft skills"? 

To sum* the trainer~eva1uator in the elaboration and design of his or 
her evaluation study: 

s - 

(1) will use sc*e words ,in the dictionary meanings of the words 

(determine* relative); 

(2) will use some words in their* technical meanings as supplied by 
educators* psychologists and others (attitude* motivation* learning); 

(3) will have-to redefine s<gne conaon words and also some technical 
words In terms of his or her special program intent and context 
(suitability* effectiveness* efficiency* rehabilitation* dropout^ 
curricula); and " - - „ 

(4) w1H have to have standards'for judging qualities of Instructional 
products* * 

to do what Is expected of tralners-evaluators* they must understand ti 
concepts of: (a) concept analysis* (b) Indicators of change^and (cj standards 
of quality of Instructional products. He will deal with these ifl the; following 

* in the order given above* 

* * 

Concept Analysis 

Evaluation and ambiguity do not go together* One must know exactly 
what is meant. by the evaluation question that Is to he answered* This meaijs 
that one must assign exact meanings to the concepts used In the evaluation 
question* For example* what Is the meaning, precisely, of self-reliance* 
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a development concept widely used In Tanzania? What Precisely Is the 
meaning of humanism which fs the bus** of. the development philosophy fn 
Zambia? What Indeed fs Integrated rural development* rehabilitation of 
economically distressed families, teacher effectiveness* library use? 
administrative support* or a dropout? 

The problem of conceptual Analysis Is the problem of definition; we 

* 

want to determine the outline and limits of the words we use* We need to 
unpack concepts * We need to specify, tte need to ask; wba,t do I realty 
-.mean? And we also need to ask; what do I not mean? 

Philosophers have developed systematic methods of how to do concept 

analysis. Sottls has suggested three different types of analyses that 

* * 

can be undertaken singly or Incomblnatfon* as part of the definition of 
concepts and* subsequently* of Indicators; 

1, Generic We analysis * Generic Is what Is general and comprehen- 
sive and which pertains to evet^tember of a class or category. In generic 
type analysis* therefore* we look for the general features of the coticePt 
being analyzed, Suppose we are analyzing the concept "development". As 
parfof generic type analysis of development* we will ask the question; 
Wha't general features must a social change program or activity have to be 
class Hied- as development? V- 

As we think of the "model" cases of development* we must also think 
of the contrary cases. For example* In the generic type analysis 'of 
development we should ask; Is economic growth In the South African 

Jonas F, Sottls * An Introduction to the Analysis of Educational 
Concepts {2nd edition), SeadTngTHassTl Addison- Wesley Publishing < 
Company* ,1970* 
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apartheid regime an instance of 'development? If not> why not? Ttte 
answers to this question should become part of our definition of develop- 

; . ■ . 

went , 

* — * , . . 

Thus, we roast work* on the one tfand, vtth clear and standard cases 

to draw the necessary features of the concept^^whe other hand* we uust 

test our ltst of features for necessity and sufficiency by thinking of 

T ' 9 ■ 

counterexamples* On the basts of this testing, we must nodtfy, quality 
and reject some features to have a workable definition of our concept, a 

2, Differentiation tupe analysts * To -differentiate ts to distinguish, 
to understand the distinct and specialized character of something* 
Differentiation type analysis may be undertaken In Heu of generic type 
analysis or in addition to It* to tighten up your definition of concepts 
even further* \ 

Tlie prior question In differentiation type of analysis is - What are 

the different basic meanings of /the concept bttng analyzed? We should 

search foY the ctomlnant standard uses of the concept (for instance* 

"development") by means of examples* We should then categorize uses Jnto 

types - political development, educational development* cultural deyetop- 

L tienf k socio- economic development. Search for distinguishing marks which 

.can be used to separate types, should follow* After this we must test the 

* * \ 

typology (categorization or classification) developed with examples and 1 

counterexamples. The scheme of relationships that exists between vaclous 

types classified may theit be Identified and a definition of tty» type In 

, which we are interested must be elaborated* % 

3* Conditions type analysis * Finally* In conditions type analysis* 

we must ask the question about context conditions that govern the use of * 
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concept such as "development*. What are the necessary conditions which 

mist be present for development to happen? Are there contexts where the 

conditions ^old but development does not occur? This might lead to a 

further search for conditions,* in the end* the set-of conditions 

established must he tested for necessity and Sufficiency. 

The three types of analyses Involve mental moves on the chessboard of 

thought. Hastening these skills Is not easy, but Is by no means 

Impossible, We learn to do such analyses by Imitating and emulating; and 

by ai:U'a1 doing and testing. The skills can be developed by learning to 

, * c 
criticize our t*m definitions and Inviting friends ami colleagues Jto find 

faults with our, analyses of concepts, * 

Let us consider >ome examples* 6 

Health education materials . The definitional Problems here may be 
quite simple. We may decide to include only teacher made .materials and 
xcJyjfe ^ll.-cofffffgrclany pro duced ma terials. We ^y_.de,cLde,Joaiicl»jd&J>_ 
posters, charts* and flannel graphs but not "only print** materials Such 
leaflets or folders. Or we may Include "only print" materials alrd none 
other, 

Dropoutv We may define as a dropout only tha* person who left the 
program on his or her owl personal decision, not the one who was failed 
and asked to leave. We may not Include one who was offered a position l n 
the course but never joined. That <s, dropouts may be separated from 
*nC"Shows*t Again, a person who came back to join the course or the 

standard from whlcn'fie or she had left, may be treated first as a "dropout" 

> 

and later as "repeater 1 *. 
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Nutrition status . Nutrition status may be based only on the 
nutritional surveillance of pregnant mothers. Uctating mothers* Infants 
and children up to five years old. Older chlldrer may or may hot be 
included. Some nutritional status reports may decide to cover everybody* 
Including the very old. 

Effectiveness . Effectiveness in one case {Example, 5 In the 
beginning of this chapter) my be defined as nothing wore than scores on 
the annual English te*t set for Form IV students. In the case of the 
effectiveness of the role of chief back In their communities, the definition 

0 

of effectiveness villi have to be developed in terms of progran objectives 
and the field context. The effectiveness may include administrative 
criteria, service criteria and political criteria, is the community 
politically sympathetic to the ruling party? Do most people feel satisfied 
with the chief? Do they see him as a ruler or as a leader? Has the chief 
Teen UneTb~estaTThsh new 1 nstTtutTons~su clTas cooperatives and prima r y 
courts In the community? etc.* etc. In other words, the concept effec- 
tiveness will have to go through the process of conceptual analysis within 
that particular program and social context. 

The questions we are asking In each case are; What are the meaning! 
of the concept we are analyzing? Uhat are not the meanings? Through 
what operations will the concept we are analyzing appear? This is what Is 
often referred to as operational izatlon . In other words* the concept is 
defined in terms of operations that have concrete existence. 

The paste purpose of concept analysis in the context of an evaluation 
stu<fy Is to assign to the concept an Invarlance of meanings. Imparlance 
means an absence of variation: thus* everyone using the concept should be< 
able to assign the sane meaning to the concept. 

124 
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Indic ators of Change 

Often the problem of conceptual analysts ts linked with a second 
problem - that of the development of Indicators, After some abstract 
concepts such as effectiveness , efficiency* rehabilitation and motivation 
have been analyzed (unpacked and their different parts specified)* another 
problem arises; How do we know that these abstract things actually exist 
In the field and are changing by some degree* In some direction? 

The essential question in evaluating the Impact of a development 

effort or the effectiveness of a development training program Is - Old 

things change? * Is "After 1 * better than "Before* 1 * on the basis of soma 

selected criteria? In other words, one must ask If a social system is In 

better shape aft 0 *" the application of some development strategy than It 

was before. Similarly* one must ask if the trainees of a development "" 

training program are better prepared to perform their tasks after their 

-t ral n 1 ftg-than-they-we re-be £o pe, - 

* 

The betterment in the condition of a society* or In the capacity of 
a development worker to undertake assigned tasks Is not* however, always 
visible to the "naked eye". Things such as individual motivation and 
commitment* problem-solving capacity* political awareness* community 
cones Iveness, responsiveness of social institutions* and the Quality Af 
life, in general, are elusive concepts, we have to look for the signs.* for 
some concrete manifestations of behavior which will Indicate that a 
community has developed! or that the trainees of a training program are 
performing better. These signs are what we call. Indicators. 

The process of developing indicators Is complex* to say the least- 
Indicators must be valid* they must be concrete ^and they must oe 
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parsimonious (that Is the Hst of Indicators for * condition should not 
be Impractical ly long). To be able to engage in Indicators research* one 
must understand the dynamics of psychological and social processes! Bust 
have sufficient grounding In logic ^and soda) science theory; ^nd have 
extensive knowledge of human behavior. We ran not* however » wait for ever 
to become experts. As practitioners and evaluators* we must learn to 
develop good enough Indicators. Conceptual analysis as discussed In the 
earlier section Is the necessary first step In the process of developing 
indicators of development or the effectiveness of development training. 

An Introduction to Indicators Research . 
\ indicators research has emerged as an Important area of research In 

\ its. own right over the last twenty years, 

\ Economic Indicators . Economic Indicators have been the oldest and 

\ most frequently used. Host of us are familiar with tfie Gross National 
, Product (GNP) per capita* the most widely used economic Indicator. Interest 
\ rates and rates of Inflation are other economic Indicators. Since the 

considerable Interest has been focussed on the development of what 
1 art called social Indicators. 1 ^ 

Social Indicators . . In the following pages* we present a few random 
examples of social indicator* that have been developed and tested by 

social scientists In recent years* to 9^ve the readers a sense of the 

\ 

current activity In Indicators research: 



Raymond A. Bauer* Social Indicators . Cambridge, Mass,: lhe H.I.I. 
Press t IQrf. 
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Example I.t; Measuring economic well-being 

Ihe following components were Identified that must be reflected In 
measuring the economic well-being of a family; 

Cash Income 

Net worth of assets owned by a family 
A family** endowment of human capital 
Ihe variability of income over time 
Intrafamlly transfers 

Ihe Impact of government expenditures and taxes* and 

Leisure and nonmarket productive activities^ 

Health Indicators as social Indicators , Health indicators Qin be 
seen as a special case of social Indicators, While most of the Indicators 
shown In the example below relate to countries, they are transferable for 
use at the regional and community levels as well. 

Example 1.2:, Health Indicators with numerical values 

A set 6f heatth^Jn)1cators along with values during the yearn 

1975-77 apj^K^p £ey>w to show how enlightening such data caa be: 



Indicators Kenya Tanzania Z ambia USSR \ US 

Ufa expectancy at birth 53 51 4fr 70 \73 



Infant mortality per 
thousand births 51 16 

Population per physician 8.840 18*490 10,370 300 600 

Percentage of population 

with access to safe water 17 33 42 - 

Marilyn Moon and Eugene Smolensky (Eds*!, Improving Beasures of 
Econowifc well-Being , Institute for Research In Poverty Monograph Series, 
New York; SIT77 Academic Press. 1977/ 
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Dally per capita calorie 
supply as percentage of 

requirement 91 36 90 138 133 

Adult literacy (which has 
been found to be a good 

health Indicator) 40% 68% 39* 99% 99* 

Health expenditures as - 

percentage of 6HP 1*8 1,9 2.6 2*7 3.3 

Per capita public 
expenditures on health 
In US* ^4 3 "y3 90 *259 

Science indicators as social indicators , SclenceTjndlcators are also 

social indicators since they Indicate the level of scler&e and technology 

in a society/ / - ' * -/ 

° ! . § 

Exawole 1,3: Scienceilndlcators / 

^ : - / 

To give the reader an idea of what kinds of indicators are used to 

develop profiles of levels of science and technology In a society* we give 

a few examples of science^tndlcators. the list is not complete: 

Research and development, 4*0) expenditures as a percent of gross national 
product (GHP) ^ 



Patents awarded for original scientific inventions 
Trade halance^Mtechnology- intensive pru&icts 
Huober of scientists produced by universities 
Students in the science track In secondary schools, etc' 

Educational Indicators as social indicators . Educational Indicators 

*■ 

are an Important part of the total profile of a society* 



^National Science Foundation! Science Indicators 1972 , Washington* 
D.C.: United States Government Printing Office. 1973. 
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ExaVfrTe 1,4: Categories of educational Indicators 

The following list of educational Indicators was developed by Cooler: 1 
ACCESS " "~ ^ 
How many and Khat Mnds of people participate In educational activities 
Retention rates In educational activities 
Catalog of existing/available educational activities or services 

ASPIRATIONS 

Description cf, needs and desires or various kinds of people 
IndlvlduaJ^selr-assessments or personal capabilities 
Description of Institutional goajs , 

// . * ' - - ■ 

ACHIEVE^' 

Whatpeople know, do, and reel 

What people have earned (degrees* diplomas, certificates) * 
^tiat is taught 

IHPACT, \ 

Consequences or having schooling 

Impact of education on social/economlc/cultural systems 

Consequence^ or not having schooling 

RESOURCES 

Capital, personnel, and material expenditures 
Quality or human resources 

1 Dennis D, Cooler , "The Development and Use or Educational Indicators** 
In Educational Indicators: Monitoring the State of Educatio n, Proceedings 
or the 1975 ETS Invitational Conference, Princeton, ffX; IducatlonaJ 
Testing Service* 1975, ^age IS, 
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Cost to benefit/effectiveness ratios 
Quality of educational climate 
fiwe . 

" Indicators of 'Indicators of Indicators '" * . 

Quite often one cycle of indicators devetPpmem my not be, enough. We 
may have to develop 'indicators of indicators 1 . Vet. even that nay be 
unsatisfactory and we way need to work on "Indicators of 'indicators of 
indicators'*. The following example should be pursued. 

* ;* 

Example 1.5: Haster social Indicators 

An American polJcy research institute 1 has suggested the following 

social indicators for developing a social report on the health of a nation; 

Individual level Societal level 

Health - 3 Wealih 

Opportunity * Opportunity 

Environment - Environment 

Standards of living Standards of living 

Public safety Public safety 

LeamJng, science and 'teaming* science and 

culture culture 

Democratic values Democratic valued 

Let us focus pn "OectucraUc values" which through a concept analysis^ 

has been found to be an Important part of die goodness of a society- * 

(Note Here Uie pervasJve and interactive relationship between "ccnfept 

analysis" and "indicator writing".} 



noept 



1 * 

'Educational Policy Research Center, Towards Haster Social Indicators . 
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HenJo Park, Ca.i Stanford Research Jnstituie, 1969 
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On pages 130 to 133t we have Included their treatment of the concept 
of democratic values to demonstrate the conceptual analysts undertaken and 
the process of moving towards, Indicators and measures. * 

Ke should note how the concept of democratic values has been broken 
down Into such dimensions as: 

(A) At the Individual level 
K Absence' of severe threat 

2, Freedom from undue social restrain* and * 

3. Freedom for personal growth 
{8) At the so c fetal level - 

1, Ho or minimal freedom 

2. Adeqljfte^for social , 
3* Synergetlc freedoms. 

We should also note* how the "attainment categories 11 *of democratic 
values In the two charts have been broken down Into subcategories and 
then into possible Indicators. 

Indicators* we should note are the types of things we can see* hear* 
sense* judge and score. 

Indicators of Interest to 

the Tralner-E valuator * 

The work of "indicator writing can not be undertaken Tightly, the 
examples above from the literature on Indicators research should help us 
understand the complexity and sophistication of the process. 

Indicator writing* however* Is not something that Is done by supermen 
and Is beyond the capacity of us tralners^evaluators. Indeed* In the 
everyday mundane settings of our work as trainers and evaluators* we will 
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Table 1 

DEMOCRATIC VALUES (INDIVIDUAL) 



Attainment Categories 



Subcategories 



Possible Indicators 



Absence of severe 
threat 



Freedom from undue 
social restraint 



Freedom from severe restrictions 
on Job* housing* schooling* public 
accommodation* voting 

Absence of strong stereotyping of 
minority by aijorlty and vice 
versa 

No strongly patronizing or mhi* 
pulatlve attitudes of majority 
towards minority 

Safety from physical threat by 
majority 



*!• Freedom of speech* press* 

assembly* privacy* due process 



*2, Freedom of religion* equality 
before law, antidiscrimination 
and antlllbel protection 



*3, Open housing, public accomfoda* 
* tlons* choice of neighborhoods 
child-bearing 



Measures of segregation 



2. Attitude studies of whites ' 
toward blacks and vice .versa 

3. Measures «f felt equality 



4, Incidence of vigilance- type 
activity* police brutality, 
_ etc*j subjective feeling of 
' fear of this type of action 

1. Nutter of law suits concerning 
these Issues* newspaper space 
devoted to them; tenor of » 
editorial cements 

2, Percentage consulted on Job 
decisions* feeling free to - 
protest Job decisions* parti* 
dpattflg In school discussions* 
feeling free to participate 

In school discussions 

*3, Measures of alienation* anomle 



Reproduced from Towards Master Social Indicators * Educational Policy Research Center* StenYord 
Research Institute*" 1969* page 



Tablp 1 (Continued) 



Attainment Categories 



Subcategories 



Possible Indicators 



Freedom for/* * 
personal growth 



1, Trust of others; overall feeling 
,of security 

Z* Openness towarfc the vlnrs, 
beliefs, needs of others 

3. Tolerance of diversity an<?of 
deviants * 

4. Responsiveness to others* views; 
willingness for others* views 
to be Institutionalized 

5. Dignity accorded to those of 
other groups 



1, Ccntefit analysts of mass 
media 

2, Various public attitude polls 

3, .Various public attitude polls 

4, "Open^nrinded - newspaper 
treatment of all viewpoints; 
acceptance of radicals and 
conservatives by business - 

$, Attitude polls - 
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Table 2 

DEMOCRAT! C VALUES (SOCIETY) 



.Attainment Categories 



Subcategories 



Possible Indicators 



Ho or minimal freedom *1, Abridgment of civil rights 
*2, Abridgment of legal rights 



Adequate for social 



Synergistic freedoms 



*3. Abridgment of human rights 



*1. Freedom of speech* pressr 
assembly, privacy* due 
process ■ * 

*2. Freedom of religion* 
equality beforf law 

3. Freedom from libel, illegal 
monopoly, illegal business , 
practices* discrimination 



1. Coincidence of institutional 
* and individual goals 



2* Attainment of full civil* 
legal* and human rights 

Reproduced from Towards Master Social Indicators * Educational 
Jtesearcri Institute* i§69* page 39* 



*1. Analysis of civil* suits* 
„ desegregation / 

2, Analysis of court decisions, 
arrest patterns 

-*3. Institutional discrimination 
in employment* housing* edu* 
* cation* etc*' 

1* Influence of public 'in deci- 
sions affecting It* voting, 
patterns* false arrests* 
availability, of free legal aid 

2* Private business* government 
„ discrimination measures 

3* 'Adherence to regulations* BBB 
activities* court records* 
data on actual minimum wages* 
trends in equal pay for equal 
work, trends In new laws 

I* Measures of Insurrectionist?!* 
riots* civil disobedience* 
drop*out1sm* alienation, 
anomie* generation gaps 

2* Number germane law cases 



Policy Research Center* Stanford 
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Table 2 (Continued) 



Attainment Categories 



Subcategories 



Possible Indicators 



3, Full participative democracy 



3. Degree to which one*man, 
one-vote doctrine holds; 
voting records, Influence of 
machine politics} ^availability 
of candidates representing 
voter spectrum* degree of 
power elitism 
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APPRAISAL CHECKLIST! GRABH1C MATERIALS 



Till* or Content of Graphics, 

ProAicer/DiSUibulor (il known) 

Seiies (\t dPPteable) 

Date (tfknownj. 



IRA^tlC 



0bJ*ttWe» btatod or Implied); 
Crlef doocrlotlon 



Formal 
□ Drawing 

Octorl • 
O Graph 
D Poster 
D Carloort 



Entry capabilities required 

Prior snbfOCI mailer knowledge 

Visual sk.us 

Ofheis 

ftATIIIG 

Simpficuy (few eJemems or ideas to catch and hoM the ailenlbn 

of the viewer) 
One main ittea (to piovide w^y) 
Relevance lo currteiitar ohfeclivi'S 
Color (allf?CIS and hokls attention) 
Verbal mtorrnation (letnlorces me ideas prosemed in tha visual) 
Learner comprehension 
legibdly (or classroom tiso 

Strong points: 

Weak Point*: 

Recomrnended ACIion 



High Medium Low 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 
ODOOO 

□ □□□□ 

□ □□□□ 



flevr6^er~ 
FosrtKxi — 



Date. 



I 



odScate. !e*<books. elo ^ar>d 
10090 (/on* CO:r»fC Strips intended 
pnmauty |o enlerlain lo drawings 
Intended loinakftimnniianl socrfll 



or nobUcai conwnenis Humor and 
sailse aro mainsiays ol the car- 
toonist's sk-N 



Cartoons aie easily omd quickly 
read and appeal lo children and 
aduHse&ke Thebesto(themcon- 
lajn wisdom as well as wit As 
such. Diey can pllen be used by 
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Validating fndlcators . Unfortunately no standard formulas can be 
suggested for writing Indicators and testing their goodness - their 
reliability and their validity. Ultimately the goodness of Indicators will 
be Proven through their testlng-by-use. 

It would always be a good Idea* however* for tralners-evaluators to 
Pre-test their Indicators through Peer reviews* They should show their 
Indicators and their 'Indicators of Indicators* to their colleagues and 
let them critique their work. 1 

The Indicators-Instrument connection . We will be discussing evaluation 
Instruments anx^thelr construction and use In the next chapter. However, 
We wish to po1nt*4ut here the clear and direct connection between the 4 
process of 1rfoV*tor development and the Process of Instrumentation. In 
the construction of tools and Instruments we merely take the next logical 
steP from Indicator development. We ask: What data or evidence should 
be collected to demonstrate. the existence^ of or change In the Indicator- 
related behavior or condition? How to collect or elicit the required 
data or evidence? What aids (tests* tapes, questionnaires, schedules, 
etc.) might be used for recording the data or evidence? (See the Figure 
on next Page.) j 

Standards of Quality of Instructio n al- Materials 



fact th 



Tralners-evaluators will also have to fa?e the P/oblems of developing 
Instructional norms (what Is a good training methodjh and standards of 
quality to Juoge a variety of instructional materials (is thk a good 
poster?). / 

i 

i 



Program «w 


Program 


Program Effects 


Indicators 


Items 


Objectives 


Actions 


and Where the 


of Effects 


for Ins'*uments 






Effects Appear 






l 
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Figure K The flow from propram objectives through indicators to Instrumentation., 
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The question of general Instructional norms has already received 
attention !n our discussions of concept analysis and Indicators development 
above. The problem of developing standards of quality to judge Instruc- 
tional products such as posters, charts* films * primers and books; ?r to 
judge overall Instructional facilities and settings Is a special one and 
deserves separate attention. 

Here we are faced with judging the Information, design* technology 
and aesthetic attributes of Instructional Items* 

Tralners-evatuators will have to learn a tot about i. .actional 
systems technology to be able to develop these standards for the * 
evaluation of Instructional products* Some "Appralsa? checklists" on 
various Items of Instructional materials developed by Helnlch, rtolenda and 
Russell^have been Included In the following pages. Special attention 
should be paid to the section on "Rating". 



'r. Itelnlch, H. Hoi end a and 0. Russell* Instructional Media . Hew York: 
John Wiley and Sons* 1982. 



> 




\ 
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APPRAlSAt CHECKMSTx GRAPHIC MATERIALS 



Title or Content of Gr*phFct_ 

P< odiJcer /Di$J**butof 0' known) 

Series (il eppkcat^e) 

Date (rf known). 



RAftHK 



Objective* (stilled or Implied]; 



Crtef rletcrtptlon 



Formal 
D Drawing 

□ chart 

□ Gtat* 
D Poster 
D Cartoon 



Entry capabilities ro<lulred 

Pnor subject mailer knowledge 

Visual skjis 

Others 

flATlUG 

Simplicity (lew elemews or ideas lo catch and hotd Iho artenlfon 

of the viewet) . 
One mam idea do provide muiyj 
fletevance lo cwttatfat nbjecltvrs 
Color (allrrcis and holds attention) 
Ve/t>aJ tfitorroaiion (ie«»tofces Hie ideas presented fa ihe vtsuai) 
Learner Comnrclien$*on 
Legibility lor Classroom use 

Strong points; 



High Medium Low 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 



Wenk Points; 



Recommended Action- 



Rewev(er_ 
Position 



Date. 



Jf Jtn Wiley 4 Sonj r*w. ywlt U« teatfc* p^nn«ion rtfrfoduca (hn Q>CcV*»t rwwn* uw <^ tHflifiiCft. Motenrta 



o^ptin4 tnerJia -newspapers f*e« 
odicats, lexibooks. etc - anrt 
r&ngo from ComiC strtos intended 
primarily (0 enrertain lo drawings 
Interred lo make unjXHltini social 



f it polilical comments Humor and 
satire am mainsiays ol Iho car 
loowsts skill 



C arloons are easily eod Qijrckly 
read arnJ appeal to cbildten and 
aduflseltke The best of them con* 
lain wisdom as well as wtt As 
such, they can pllen be used by 
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^APPRAISAL CHECKLIST! AUDIO MATERIALS 



TJtto. 



Series (if applicable). 
Dal* (H kncwn) 



Pflce_ 



Objective* tstdted or Implied): 
Oriel description; 

Cntrjr cnpnbllW<»a required: 

-Pt*x subject matter knowledge 

-At>d*o Skiffs 

-Other 



RATING 

Accuracy 
Sound fjuafity 
Student invotveinenl 
Interest level 
V6cflbu'a*y fovel 
Overall value 

Strong potnlet 
Wcflk point st 

flecomme*>ded action. 



High Medium Low 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 



formal 
—Record 

Reetio reel 

Cassette 



Speed Time 

ipm min 

_1>s 



Hevtewcf- 

F\lSition 

Oate 
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APPRAISAL CHECKLIST* FILM 



Tltte- 



flfoducef/dtstribulof- 
Dale (il known) 



'Length. 



Subject area(s) 



RATtMQ 

Likely to arouse siuctaii intetest 
Technical fjuafciy 

OpfKMiiiifly for viowet parhcipalton 
Rcrcvartco lo cunictfkjm (or teaioing iask> 
Acciwacy of mlwmalion 
Scopo ol coittOnl 
Of (janizalton ol content 
N Stu lent compiehenston 

Strong point*! 
Weekpofnlu 

Recommended action 



H>gh Medium Low 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 

□ □□□□ 



Format ^—16 mm 
8 mm 

_jother 



Objective* Ut* ted or Implied): 

Brief dcacrlpllcn {Include presentation »tWet ontmetedt 
dramatic i etc .J; 

Cntry capabilities required! 

_tanguage ability 

—prior subject mallet knowledge 



color 

Wadtfwhiie 



Reviewer- 

PO$i1*QP — 



Dale. 
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APPRAISAL CHECKLIST* TELEVISION 



Strict im* 

Program title (w nun*ei)_ 
ProdUcOf/dtSlributo 



ProftxAkn dale : r Program length- 

Intended audience/grade level Subject area 



h 

Objectl G*t»t*tHoMmp11*(t)t ^ ' 

Brief* Jicrlptlom 

Cntry *ap*»MMtfet fftqiilreth 

— prkx knowledge 

— reading abW/vocabutary 

— math at>Hity 

«»gn Medium Low 

Lftefy to arouse student interest □ □□□□ 

Technical quality 

Provides meaning fut view participation □ □□□□ 

Objectives relevant to currteuf ar needs □ □□□□ 

Focuses ctearly on objectives 

Evidence ot effectiveness (e g . field-test lesufls) □ □□□□ 
Teacher^ role deany bvficaied □ □□□□ 

Psrovide* guide tor dfecus^jo/ToQow up □ □ □□□ 

«tron9 polnUt 

«i *. ^Ii -JM Fleviewei- 
Weiifc pain tit 

Position 

necontroendod action Oale 
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BEST COPY AVAILABLE 
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APPRAISAL CHECKUSTt SIMULATION/GAME 



THIo- 



Putobcatton dale- 



Number d player s« 



Speciai eqiiP*nent or facilities needed, 
tnienoed audience^rade level 



_0 Has gammo, features 
(compel iiloa sco*u>9) 

O Has simulation leatures 
playing) 

Playing linia . 



Subject are*- 



Objective* Utattd ct Implied)! 
1 DtfletdHCtlPtiom 

Cnlry CQpAbllUlea f eqqttod 

_ piiot fcftovviedge 
— reading abiMy, 
_maih atxhiy. 

RATtrtQ 

Iskeiy 10 arouse sludem <nter est? 
' Players practice meaningful skills 
(Game} Wmrwig dependent on ptayer actions (vs ctiance) 
(SimuJaiion)Vaiidtfy ot gamu model (reaiisiic, accurate) 
Technical quahly (dutafrtoy* attractiveness, etc ) 
Evidence ol effectiveness (e g lieid lest fesuBs) 
Clear ejections (of conducing game? 
layers' instructions dear ano concise? 
Debating guide included? 

SUong point* i 
Woak polntsi 

.ttecomn:2<xJcd aciion : 



High Med turn Low 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 
P □ □ □ □ 

□ □ □ □ □ 

□ □ □ □ □ 

□ □ □ Q □ 

□ Q □ D □ 

□ □ □ □ □ 



POSilKXl— 

Oaie 



John Wsj 4 S«* (pt grjiiina r<t*Ji;> ptc<twj«owicpioaix* ^Ct*vi^('o< (WionaiutiJOfity (ltww.n MdenSj 
wdRuiiCO ilftirwCliONAL MEDIA Oili 
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Stwfnary 



To go from the stage of having an "evaluation question" to the 
development of tools for collecting "evaluation data"*, tra^ners-e valuators 
must cross the two Important bridges or concept analysis and Indicator * 
writing. Concept analysis Involves three Interrelated analytical * 
strategies! generic type analysis* differentiation type'analysls and # 
conditions type analysis. The basic questions are: What does the concept 
mean? What does the concept $ot mean? What operations lead to the 
emergence In real-life of the concept and under what conditions? 

Indicator writing Is an extension of the Idea of concept analysis, 1 
The questions now asked are; What concrete entitles and behavioral data 
(things -we can see. hear* touch* sense* judge and score) will be considered 
as evidence that the concept In question exists In reaT-Tlfe* or Is 
changing In some measure In some direction? The continuous link from 
program objectives, through program actions* program effects.and where 
those effects appear* Indicators of effects to Hems for different tools 
and Instruments Is Indicated. The special need for standards of quality 
forjudging Instructional materials has been discussed. 

Things to Do or Think About 

1 Examine the definitions of some terms given below. 

(a) Which ones are definitions of common words that you are ready 
to accept as universal? 

(b) Which ones are delfnltl ons of technical words that* again* you 
will accept as generally applicable? 

(c) Which ones are dellnltlons of technical words but which must be 
modified through the process of concept analysis within your particular 
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program and social context? 

Truth : Conformity to fact or reality* to rule* standard* rootlet • 
pattern or Ideal. Faithfulness to the facts of ( ature» history or 
life; 

Learning : Permanent change in behavior that is the result of past 
experience either produced Incidentally or through institutional 
'learning through teaching. 

Primary education : Those years of study during which no differentia- 
tion Is introduced either In the forms of optional subjects or in the 
streaming of pupils towards different types of Institution or 
education. , 
Dropout : Person who leaves school or college before completing the 
course of study. - 

2. Undertake the concept analysis of the concept of "self-reliance 4 * in 
the Tanzanlan setting, 

3. Develop the indicators of "training effectiveness* 1 for a hypothetical 
training course for rural health workers. 

4. What would you want to see in: 
a good primer 

a good rural newspaper 

a good poster or 

a good teaching film? 



/ 





CHAPTER X 

EVALUATION 100LS AHO INSTRtfltEHTS , 

* " ° 

In Chapter I (What Is Evaluation), it was Pointed out that while 

"evaluation" and "research" differ significantly from each other in regard 

to their objectives and functus* the evaluator and researcher do share 

their general approaches* instruments and tools for data collection. 

later, in our discussion of the two basic paradigms of evaluation and 

research (Chapter VIII; Models of Evaluation), we Indicated that the 

scientific paradiojn and the naturalistic Paradigm are rooted in two differen 

wortdviews and demand two quite distinct methodological aPProat, * studying 

the world and maHng Knowledgeable statements about this world. Consequently* 

the kinds of instruments chosen by evaluaters using the scientific Paradigm 

are different from those chosen by evaluaters using the naturalistic paradigm, 

The scientific Paradigm typical fy demands structured instruments of data 

collection, Within the naturalistic paradigm, the evaluator himself or 

herself is the primary Instrument: the approaches, the tools and the' 

Instruments in use are unstructured, The following display should clarify 

the choices of tools and Instruments made by evaluators and researchers 

within the two Paradigms - the scientific and the naturalistic; 
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In the following* we will deal with the various tools aqd Instruments 
recorded in the display above. 



Scales of Measurement 



Measurement Is an Important part of evaluation. We often need to go 
beyond the crude comparisons of H good<*better~best* and"b1g*bigber'*bi9gest* H 

To do this we need. standard yardsticks wftlt which we can take the measures 

1 

we want; and can state how much of a difference exists between itwo entities* 
and in what direction* 

- Unfortunately, In the social sciences we do not have the benefit of 
such tools as micrometers* carbon dating and atomic clocks* Dur measures and 
yardsticks are often quite crude, We need to understand* however, the 
nature of scales that are available to us; and we need to understand their 
possibilities and their limitations. 

The nogjpaj scale . The nominal scale does not really measure* It only 
nominates objects to categories'. The classification of adults in a cwiittfnity 
Tnto males and females* and assigning then numbers (I for males* Z for 
females)* will be an example of using a nominal scale. 



o . 148 



147 



Ite need to understand that numbers used In the nominal scale mean 
nothing about the value of categories* except to show that they ire different. 
]n the above example* 2 (for females) is not twlcs of I (for wales). The 
numbers 2 and l» in this particular context* can not be added or subtracted 
frov each other In any meaningful nay. - 

The ordinal scale . The ordinal scale Introduces ordering to the 
nominal scale. The categories can now be ranked in an order of succession 
as "first* Second and Third** 1 or "Good* Medium and Poor/' 

The ordinal scale^ again, could be assigned numerical values: for 
examplei 5 for Good, 3 for Hedlum and X for Poor.. But, once again » 5 is not 
five times 1 in terms of the scale* nor Is 6 (two radiums) better than 5 
(one good). 

The interval scale . The Interval scale* as the name suggests, has 
Intervals which make mathematical sense. On a meter rod, the difference 
between 3 and 5 centimeters Is the same as the distance between 53 and 55 
centimeters. 

Scores on an achievement test are in reality ordinal data, but we can 



often treat It as if it was interval data. We can say that 6 made twenty 
points (or twenty intervals) more than A. However* if 0 had made 40 points 
and A had made 20* we could not say that 6 ts twice as good as A. To be 
able to make that kind of statement* we will need ratio scales.- 

The ratio scale . The ratio scale* In addition to being an interval 
scale* has an absolute zero. This means that 25 Is 5 points more than 20* 
and that 60 Is three times as good as 2D. Thus* the ratio scale permits us" 
to work out ratios and proportions. Two meters is twice as long as 1 meter. 
Something can be twice as hot as another. 
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We need to keep the properties of various scales of measurement In view 
as we deal wHS data from our various evaluation tests, tools and Instruments* 

r / 

Tests 

4 

Anyone, who has been h to school has been subjected to tests (or exams as 
they are popularly called.) Tests are a, usual tool of the. evaluate*- working 
within the scientific paradigm. Evaluators working within the naturalistic 
paradigm o^y avoid formal testing* but they can not* however* avoid judging 
and selecting, in judging and selecting* naturalistic evaluators will * 
perhaps use observation* of performance-related behavior within the relevant 

social context Instead of formal tests. ' 

j 

Tests* or achievement tests as ttjey are often called, are tests of 

T 

knowledge* skills and performance. Tests may be made to measure knowledge 
In arithmetic* biology* nutr1t1on~or animal husbandry; research skills* 
diagnostic skills or graphic skills* or actual performance in a role*. 

% Tests can also be used to measure aptitudes (natural or acquired 
^abilities or bents of mind). In fact an aptitude test cap be seen to be a 
special kind of achievement test. ^tf^ 

t ^valuators may sometimes be Interested In testing attitudes (value 
dispositions and opinions). Attitude testing will be discussed later a? 
part of questionnaires and Interviews. 

I faying gone through lot of achievement t.-stlng In our lives; and* 
Perhaps, having ourselves w> n and administered tests as teachers and 
tralnerSf^we might think of tests as relatively simple to make* to administer 
and to Interpret. This Is not really true. There^are many complexities 
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Involved as the discussion that follows should show. 

Standardized Honn-fleferenced Tests and 
CrlteMon-fleferenced Tests 

Tests may be made Tor one particular group (community health workers 
'under training In a special workshop) or Tor a large regional or national 
population (all VIII grade students in Kenyan schools or even East African 
schools). 

-in the first ca<e» the test will most likely be designed to* Measure 
whether the cofzrounlty health workers have learned most or what they were 
taught within the special workshop that they had attended. The criterion of 
success may be a score of at least 80' out of. the possible 100 on on achieve- 
ment test specially designed for that group. This would be *n exa*»Pie of a 
criterion-referenced test. 

In the second of the two cases above* the test will oost^Hfcely be 
designed to measure how well a student* a class, or a school is doing in 
comparison to other students* classes and schools tested on the same test of 
VIII grade mathematics or English 0 r civics. To be able to make those 
comparisons, we will have to have norms - how is an average Vffl grade 
student supposed to perform on this particular test, tihen these uorms.du 
become available, the test becomes norm- referenced and standardized. 

The process of standardization of tests for development of norms is 

itself quite standardized now. We do not discuss it here Because trainers- 

* 

evaluators will most often he dealing with criterion-referenced tests. 
Those are tte tests we will fucus upon in the following discussion. 
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T tailing Qjx l e_c tWes ,_ _o j*d 

Training and testing objectives should match with each other. It 
would he patently unfair to test trainees on things they were never taught. 
This means that the test writer should have available to him or her a clear 
and detailed statement of the Instructional objectives of a training course, 
to be able to make a test that will measure the Impact of the course* 
effectively. 

Professors Benjamin S. Bloom, D. R. Krathwohl and their associates have 
developed taxonomies of Instructional objectives that should Interest both 
trainers and te*t makers. The basic outlines of their taxonomies are 
reproduced below: 



1NSTRUCT JOHAL OBJECTIVES \H THE COGNITIVE DOMAIN 

t.QQ Knowledge 

t.10 Knowledge of specifics 

1.11 Knowledge of terminology 

1.12 Knowledge of specific facts 

1.20 Knowledge of ways and means of dealing with specifics 

1.21 Knowledge of contentions 

1.22 Knowledge of sequences 

1.23 Knowledge of classifications and categories 

1.24 Knowledge of criteria 

1.25 Knowledge of methodology 

1 30 Knowledge of the unWersals and abstractions In a field 

1.31 Knowledge of principles and generalizations 

1.32 Knowledge of theories and structures 

?.00 Comprehension 

2.10 Translation 
2.2*> Interpretation 
Extrapolation 

\ 

\ 3.00 Application 



0. 5. Bloom etal , . Taxonomy of Educational Objectives^ Handbook I : 
in1_t1ve,gotna_1n. Tie* YorF~ DavlTflcKay, 1956. \ 
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4.00 Analysis 

4.10 Analysis of elements 

4.20 Analysis of relationships 

4.30 Analysis of organizational principles 

5,00 Synthesis 



5 .TO Production of a unique communication 
5.20 Production of a plan, or proposed set of operations 
5.30 Derivation of a set of abstract relations 

6.00 Evaluation 

6.10 Judgements in terms of internal evidence * 
6,20 Judgements in term* of external criteria 

INSTRUCTIONAL OBJECTIVES IN 1IJE AFFfcCTlVt DOMAIN* 

1,00 Receiving (attending) 

1.1 Awareness 

1.2 Willingness to receive 

1.3 Controlled or selected attention * 

2.00 Responding 

2.1 Acquiescence In responding 

2.2 MllUngness to respond 

2.3 Satisfaction In response 

100 Valuing 

3.1 Acceptance of < value 

3.2 Preference for a value 

3.3 Couinitment 

4.00 Organization 
\ 4.1 Conceptualizing a value 
4.2 Organizing a value system 

5.O0, Characterization by a value or value complex 

5.1 Generalized set 

5.2 Characterization 

The test writer should not confuse the cognitive with the affective, or 
the ability to synthesize with the s1inp1e\nowledge of unlversals and abstrac- 
tions. Ue should realize that learning of information does not ensure real 
comprehension; and comprehension does not automatical ly lead to the ability 

— : \ 

c 0, R. Krathwohl, et a) M Taxonomy of Edu ca tional Objectives. Handbook 
il: Affective Domain . Hew York: David HcKay, 1964. \ 
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to apply* analyze and Judge. Similarly* ft Is possible to be positive 
verbal l y to particular entity or a position without genuine commitment; 
and to have a set of discrete values that do not add up to a systematic 
and organized value system. He simply can not teach one th Inland test for 
another. 

ChooslPp the Test Content 

It Is obvious that one can not test everything that has been taught. 
One will have to take a small sample of all the knowledge taught, to be 
Included In a test. 

The saWle of knowledge to be Included In a test should be developed 
systematically from a detailed and comprehensive description of the 
subject matter taught. The two taxonomies presented above should be used 
Tor the description of subject matter taught: What factual knowledge was 
taught? What general principles and generalizations were communicated? 
What diagnostic skills and abilities to aPPly and transfer to other 
situations were underlined? What change In attitudes and values was 
reinforced as part of tt.e resoctallzatlon for the new role? 

Based on this comprehensive description, a sample of knowledge and 
values should be selected for test making. 

TrEgsj af Tea , t _I terns 

A variety of test Item can be written to be Included In an 
achievement test. 

True/ fa lse. A statement Is written and the respondent Is asked to 
check It as true or false. 
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Example . 

Groundnuts and vegetables are 

body-building foods. T/F 

(Answer: True) 

True/false Items are comparatively easy to write* These are. however. 
*of limited use In testing for depth of understanding. The advantage of 

easy -seo rtng-ts-ba lanced by a disadvantage* Respondents feel encouraged to 

guess answers when they do not really know the answer* As they make 
guesses, they have a 5Q:50 chance of being right. 

Short answer and completion Items, . As the name suggests, these Items 
require a short one- or two-word answer or the filling In of a blank* 

Examples . 

what do spittle and rubbish breed? 

(Answer: Microbes, ) 

239 
-143 

= (Answer: 96) 

The manometer of the sprayer shows that It 
has p 

j t (Answer: Pressure) 

Short answer and completion Hems have to be written carefully so 

that more than one Interpretation of the quest Ion/ incomplete sentence Is 

not possible. The wording of the item should elicit the Information 

specifically required* 

Hatching , Hatching Involves pairing of Items from two different 

sets or columns because of their similarity or correspondence according to 

some rule or relationship. 

15U 
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Exarnpje. 
(Column 1) 
(1) Ecology, 




{Colume 2) 



The Pattern of Interconnected food chains. 



{Z) Predatloi 



The taking In and using of organic 
food for energy* growth and replacing 
cells. 



(3) Nutrition 



The study of hoti living things relate to 
each other and to their nonliving 
environment. 



A relationship between two kinds of 
organisms In which one benefits by killing 
and eating the other. 



Hatching Items should be fcePt relatively short, Note that there are 
three choices under Column 1 and four choices under Column 2. This 
Insures that matching will Involve deliberate choices In all cases under 
Column 1. 

Jf a choice under one of the columns Is usable more than once* make 
that Information available to students as a part of the question* 

M ultiple-c hoke* MultlPle-cholce "items are the most versatile and 
effective form of test Items. A mul tfple-cholce item has a stent* followed 
by multiple options from which one or more could be selected* 

Ex ampJ^* 

A farmer shovld do early weeding of his cotton croP: [Stem] 

(a) So that the cotton Is not chocked 

(b) So that weeds do not consume up the plants' food 

(c) So that cotton gets enough air 

(d) So that cotton has access to light 

(e) So that cotton gets enough water 

(0 Because weet* could breed dangerous Insects for cotton 
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(g) To allow better growth of cotton 



(h) To get a good cotton yield 



[Option*] 



Host of the options above are correct. Choosfng the right options 
and leaving out the Incorrect ones will be like writing a short essay on 
the advantages of early weeding of the cotton crop. 

Typically* multiple-choke Items nave no more than four or five 
options, unlike the Item above which has eight options. 

Essay * This Is the easiest type of test to write and the most 
difficult one to score. When essay questions are carefully written, 
specifying exactly what Is required, essay questions do provide the 
students with opportunities to analyze* synthesize and evaluate subject 
matter content. Objectivity of scoring of essay type questions can be 
Increased If teachers themselves write model answers to their own essay 
type questions and* then judge student responses according to the model 



Slmultatlons . Simulations of various kinds provide exciting teaching 
and testing possibilities, various types of "In-Tray/Out-Tray" slmultatlons 
can be designed to test the performance abilities of trainees In life- 
like decision-making situations. 

Pre- testing Tests for improvement 

Good test 1 tents have to 'test what they are supposed to test; and 
should be well -written so that they communicate Ute same one meaning to all 
readers clearly and unambiguously. 

Item writing takes time, patience and skills. Hlth time and patience, 
skills can be developed. One thing that test writers must do Is to 



• answers . 
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pre-test their tests* and go through careful revisions of their tests on 
the basis of their pre-testlng. 

After a more wide-scale use of a test 1" an evaluation stuo> t the 
test should be" revised again. Even If you W111 never use ft again* the 
revisions wlU train you for better test writing for future-evaluation 
studies. 

i/ 

Time Tests* Power Tests and ( 
Other Considerations Itl Attain Isterl no Tests 

T ests should be administered so as not to make respondents afraid 
and anxious - what Is called "test anxiety* can becometa serious problem, 
Indeed, within developmental settings* where we deal with adults (and 
also with government functionaries)* we may find that we want to give a 
test but the adults concerned do not, want to take the test. Sometimes a 
few test Items nay have to be hidden In an opinion questionnaire or an 
Interview schedule. 

When administering a test* the respondents should be comfortably 

*. 

seated and Instructions In regard to how to complete the test should be 

i 

fully explained. 

i 

Finally* tests can be time tests or power tests, *T1me tests have to 
be completed within a Particular Period of time: 45 minutes or an hour* 
for Instance. At the end ol this time* test papers are collected whether 
or not these have been completed. Power tests are given to determine 
how much the respoodeots hau* learned (and not how fa*t they can answer 
questions). In a power test* there are many more test Items thdn there 
are In a typical time test* and time Is allotted generously to students for 
completion of the test. * 
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Questionnaires 



Questionnaires are a set of questions* systematically arranged on 



.paper to be asked from a group of respondents. 

Spaces for recording answers are provided Within the questionnaire. 
Questions may tie open-ended, which means that the respondent will formulate 
the answer in his or her own words. Or; the answers may be structured* 
requiring the respondent to make multiple choices or to check on scales 
built Into the questionnaire. 

Questionnaires are often distributed by hand or sent by mall. When 
dealing with Illiterate respondents or to ensure high rate of response* 
the evaluator may administer questionnaires In person. In such a case the 
questionnaire becomes a "structured Interview/ 1 

Questionnaires should be short and well -designed. Instructions 
should be clear and easy to understand. A short introduction should 
provide the purpose of the questionnaire and explain how the data provided 
by the respondent will help the respondent or others in the comounlty. 
Anonymity of respondents should be ensured. 

In-the preceding section on tests* we have suggested that tests are 
test of knowledge. Questionnaires ateo test knowledge but it is the 
''particular knowledge" of a person that questionnaires seek to find out 
about. Jt Is nut the general knowledge of subject matter hut the private 
knowledge of a person - information personally available, his or her 
percept Ions /attitudes and opinions of various kinds. (As we have 
Indicated 1i the preceding section* some achievement test Hems may 
sometimes be hidden in a questionnaire.) 




Local Adaptation of Available Questionnaires 

Questionnaires can not be standardized (as noro- referenced tests), 
Tralners-evaluators will be making their own questionnaires to suit the 
special social and program context of their evaluation studies. However* 

Questionnaires on slml lar subjects developed by other tralners-evaluators 

J 

elsewhere may be adapted to fulfill particular needs* tony Questionnaire * 
Items may be possible to lift and use with very little rewriting. 

Ty pes of Questionnaire Items 

All the types of Items that can be used In an achievement test* can 
aTso be used In a questionnaire: true/false* short answer or completion* 
matching* multlpTe-cholce* short essay and even simulations. The only 
difference Is that we are now eliciting "private knowledge/' perceptions* 
attitudes* Opinions and Judgments, {Refer to the discussion on "Types of 
Test items" under Section I above*) 

Making Scales for Recording Attitudes 
and Opinions 

In designing questionnaires which Include lots of attltudlnal and 
opinion questions* the evaluator wilt have h> design scales to record the 
subject's responses. 

Sometimes these scales may be In the simple form of 
Yes Undecided Ho 

i J L 

or 

Agree Don't know Disagree 
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To wake the scale pure sensitive, additional ordlnates may be added, 



as 



Strongly Agree Don't know Disagree Strongly 

agree disagree 



Robert 8. Jones' made an opinion survey of military officers in regard 
to what they thought of their "military assistance training courses'* and 
'used a scale with many 1 dimensions. (The scale below also shows. In per- 
centages, the responses actually received.) 

MILITARY ASSISTANCE COURSES IN GENERAL 





{Based on the 


Courses You' 


lave Taken) 




Creative 




22.2 




14.8 


14.8 


14.8* 




Uncreatlve 


Organized 


26.9 


34,6 


1M 


3.8 




15.4 




Disorganized 


Hard 




3.8 




26,9 




ii^ 


7.7 


Easy 


Relevant 




34.6 


26,9 


19,2 


7.7 




3,8 


Irrelevant 


' Flexible 




1L.5 


26.9 


15.4 


11.5 


23.J, 


11.5 


Inflexible 


> Practical 


7.7 


30.8 


26,9 


19.2 


1L5 




3.8 


Impractical 


Exciting 




1LJL 


30.8 


19.2 


19.2 


1^.5 


7.7 


Dull 


Strong 


_L8 


15.4 


34.6 


23.1 


7,7 


7.7 


JV7 


Weak 


Active 


3,8 


19.2 


14.6 


19.2 


15.4 


7.7 




Passive 


Demanding 


3.8 


7.7 


26,9 


19.2 


15.4 


19.2 


_2.,7 


Undemanding 


Scientific 




15.4 


19.2 


3B.5 


11.5 


7.7 


_7._7 


Artistic 


Involving 


3.8 




3Q.5 


ISA 


11.5 


3.8 


-Li 


Alienating 


Objective 


JL8 


LLi 


34.6 


19,2 


15,4 


11.5 


3.8 


Subjective 



'Robert 8. Jones. T he influential in Uniform : A Study of the Train - 
ing, Attitudes and Role Perceptions of the Uni ted StateVffilUary Assistance 
Officers as Change Agents In Developing Countries ^ Doctoral Dissertation. 
Indiana University. Uff. 
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Modifiable 
Motivating 



/ 

/ 
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7.7 IKS 26.9 30.8 _7 r 7 11,5 3.8 Unmodlflable 
3.3 19.2 26.9 34j[ _7il -Zl- ML Alienating 



This scale Is multi-dimensional since It uses many bipolar dimensions: 



creatlve-tmcreatlve* organized-disorganized* hard-easy* etc. Also* the 
scale Introduces seven ordinal points along each dimension of the scafe. 

Jones ^as collated data In percentages; What Percentage of respondents 
chose each ordinal point on each of the dimensions of the scale? As we wljl 
discuss later* ordlnates can be given numerical values as follows: 
Creative Uncreatlve 

_ * * t I I I i_ 

*3 \ +2 *t 0 -1 -2 -3 

i t 

Such scores can then be aggregated./ One has to learn to be careful 
about assigning numerical values or we/ghts to responses on such scales* 
however. [ 

i 

W riting Good Questionnaires / 

Good questionnaires are made wltjj clear objectives In, view. Th<sy 
ask what the tralner-evaluater needs to know* avoiding Unnecessary questions. 
But the Important questions are not forgotten. Standard demographic 
Information such as sex* age* nccupatlon, Income* etc* Is always asked to 
be able to Interpret responses received. 

Hem writing for questionnaires offers an additional set of problems 
since (1) they may ask foy private knowledge that the respondents may be 
unwilling to 'part with) and (2). they elicit opinions and attlcudes chat 
the respondents may be unprepared to share honestly. Attitudes In regard 
tn family planning* Inter-marriages between people from different tribes* 
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and taboo foods day not be honestly expressed. The respondent may supply 
"socially acceptable" responses. They may tell the traf ners-evaluators 
what they "want £o hear rather than what the respondents actually think. 

To solve some of these problems, writers of questionnaires may make 
the Intent of an Hem less direct and may ask the same question in 
different ways within the same one questionnaire, 

Once again* pre- testing (^questionnaires Is Important before 
administering Jhem On a large scale as part of an evaluation study, 
pre- test I fig will bring out many problems In the questionnaire. 

Let us look at some examples: 



Such 



t Items 

A df strict officer is asked: 
After Information has been 
communicated to the chiefs/ 
assistant chiefs In your 
area* how Is this acted upon? 

A community level nutrition 
worker fsjasked: Nhat do you 
engage Injuring your home 
visits? h 

A subject is asked; Do you 
attribute your friend's 
failure to* laziness? 

A subject Is asked: Do you 
think you were In good health 
during the period of the 
training course? 



An extension worker under 
training is asked; Was your 
visit to the farmer useful? 



Comma nts 

Can the district officer really 
tell? Wouldn't It be better to get 
this information from the chiefs 
themselves? Aren't we asking the 
wrong respondent? 

Isn't this too general a question? 



What Is laziness? Do we all mean 
the satoe thing by the word laziness? 
This Is atfriguous. 

Do the subject and the evaluator 
understand the saiue thing by health? 
What. If the student has not been 
too well, but never too sick to miss 
classes for long? Shouldn't we ask 
the question in terms of days missed, 
because of sickness? The question 
Is anblguous In Its present form, 

Useful to whom? In what way? On 
Ihe basis of what kind of evidence, 
us,ing what criteria? Amoiguous* 
again. 
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The headmaster of the school 
Is asked to judge the student? 
teacher's commitment to work In 
term of: 

unsatisfactory 

— below average 
average * 

— above average * 

— outstanding 

A local extension worker Is 
■asked by the evaluator: Are 
locally made audio-visual 
materials better than those 
produced elsewhere 7 



A cooperative assistant at the 
comnuntty level Is Wed by the 
. evatuator: Are youjhfghly 
motivated to make we best 
contribution to your work7 

An adult education officer fs 
asked: tfow many of, your 
earlier students still practice 
reading sk1Us7 

foany of these problems may be caught In the process of pre-testlng 
of the questionnaire. With practice, item writing for questionnaires 
will surety improve*, 

ill 

Interviews. % 

I 

Interviews are used by evaluators both for scientific Inquiry and 
naturalistic Inquiry, [n the context of the scientific paradigm, Interviews 
are structured or semi -structured {basically a structured Interview, with 
some probing questions Introduced to seek further explanations), I it the 

context of the naturalistic paradigm, Interviews are unstructured and / 

t 

are conducted In depth ror developing "thick descriptions'," . 



Ifow to ensure that the evaluator 
and the^headmaster mean, the same 
thing by the word comnkment7 Do 
we define commitment fn terms of 
punctuality* or carrying an overload 
of work, or offering tutorials to 
weak students7 llow 111 the head- 
master come to acquire the *,nowledg£ 
4 on which to base his/her judgments? 

' v Does "elsewhere" mean fn another 
locality 7 National headquarters 7 
A commercial producer7 Does better 
mean better In production values or 
fn terms of instructional retevance7* 
There are many ambiguities* 

Wouldn't most of them say "Ves,"7 
4 Isn't It a loaded question? 



Is this question answerab1?7 
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As we have Indicated before, structured questionnaire when admin- 
istered in person become structured Interviews. The structured interview* 



display of Items as doss the structured questionnaire. (See Section II 



But since Interviews are conducted in face-to-face situations* they 
pose some additional problems and challenges. The interviewee must be 
motivated to give the Interview and to Invest the time required for 
completing the Interview. The Interviewer should be able to establish 
trust and rapport without influencing the responses of the Interviewee. 
In rural settings of developlrtq countries* it may not be possible always 
to take the interviewee (especially a female Interviewee) aside for a 
l*ng private conversation. The Interviewer should assure that an 
Individual Interview with a young mother does not become a family Interview* 

Sometimes faaily Interviews n>ay b<> Just the thing we want. But then 
we should plan and work for a family interview. The point Is that an 
Individual Interview should not get confused with a family or group 
Interview, 

it Is also possible to use more than one interviewer in conducting an 
interview* A chief pay be interviewed about his work by a full panel of 
interviewers. 

In-depth Interviews for 
naturalistic Evaluations 

Interviews conducted as part of naturalistic evaluations are not 
structured. The interviewer starts not with questions but with theras, 
Afier establishing the general boundaries of the subject of the Interview* 



therefore* has the same problems and concerns of design* Item writing and 
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the naturalistic Interviewer | e ts the Interviewee take over. The Interviewer 
listens* With Interest and sympathy, encouraging the Interviewee to go on, 
to explain further* to come back to thi* point* to choose* to judge and to 
take positions. Such Interviews typically take several hours. 

In-depth Inter/lews may be recorded by the Interviewer 1m His diary In 
the Presence of the Interviewee or Immediately after the Interview when the 
Interview is still fresh In the Interviewer's mind. Tape-recording the 
Interview may be preferable, If the interviewee willingly permits. 

IV 

Field Observation 

Field observation* again* Is a data collection strategy that can be 
used both within the scientific and the naturalistic paradigms. Once 
again* field observation within the scientific paradigm may be sampled and 
highly structured. JfltMn the -aturalfstlc tradition* Held observation 
will be unstructuredjand leisurely. He may make participant observation 
or nonpartfclpant observation. Observation can be both overt and covert. 

Eva I ua tors wantj to make Held observations to get a direct sense of 
the reality without ^n Intermediary having to see and Interpret It for us. 
Observation Is not* r^owever* a matter sL^ly of opening our eyes and ears 
to peoPle In reaMlfe situations. We have to train our eyes and ears and 
must learn to record o^r observations. Claries t check-lists* maps and 
diagrams* schedules* soclomy tries, rating scales* and cameras can all be 
used fnr recording observations. 

Observation schedules are by no means easy to *Hte and a variety of 
errors are common In tnetn: 1 
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Comments 



Is the student-teacher audible 
enough to pupils sitting at 
the back of the class? 



Does the student- teacher speak 
with confidence? 

What economic status do the 
loanees have? 

How did the loanees use the funds 
they obtained from the, coopera- 
tive society; nia/rled second 
wives* paid children's school 
fees, engaged In heavy drinking, 
or bought new clothes? 

An observation schedule seeks 
to observe : 

— attitudes of people 
before the public meeting 

, starts; and 

— attHudss of the people 
during and after the 
public meeting. 

Does the cooperative society 
keep the books required under 
the law? 



Can this be "observed* 1 ? Or do we 
have to ask the backbenchers about 
It? Or should the ©valuator walk to 
the back of the room and listen? 
The Item needs clarification. 

What should we look for when observing 
a display of confidence? 

Can one "observe* 1 economic status as 
such? 

How can we observe this history of 
behavior in a visit or dt/rfng a 
short period of observation? Such 
Information will have to be ( 
collected through alternative mean*. 



Is this possible to do? Do attitudes 
change In the course of a public 
meet I no? Do attitudes short on 
people T s faces? 



Okay* but Isn't this a natter of an 
audit rather than observation? 



Records and Documents 

Records and documents are important sources of data for the evaluator, 
Ihe analysis of records and docuutents may be quantitative (suited to the 
scientific paradigm) or qualitative (Suited to the naturalistic paradigm). 

The basic question asked from records and documents Is "who says 
what, to Ufcom, How. and with what effect and Why? The techniques of 
analy&H are aynreyatlon-lntegration method (quantitative), trend analysis 
(quantitative) r content analysis (both quantitative and qualitative)* and 
case-purvey aggregation (qualitative). 
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We will have more to say about content analysis later In Chapter XIV. 

The Ethics of BuyJng Dat a 

the question has often been raised: Should an e valuator Pay his or 
her respondents for Participation in an evaluation study? There Is no 
IfnpTe 'yes* or *no* answer, 

Knowledge production Is a social function! and In the case of an 
evaluation study* the social use of evaluative Information can often be 
quite clear both for evaluators and respondents . If the evatuator Is 
working In behalf of the government o** a non-Profit making voluntary 
agency* It Is Public Interest which Is being served by the evatuatlon. 
The respondents* as good and concerned citizens* sliouTd freely Participate 
In the evaluation study. 

If, however, a subject Is put In a position to chose between working 
on a construction site for the day or Participating In your evaluation 
study* you should then Pay to compensate for the wages lost by the 
respondent. But where the time Invested could not have been sold for 
wages anyway* payments should not be made* H 

Evaluators typically use achievement tests, questionnaires, Interview 
schedules* observation schedules, and records and documents as Uplr 
sources ol data. The e valuator using the scientific paradigm uses 
structured tools and Instruments and formal approaches. The evaluator 
using the naturalistic Paradigm makes himself or herself the Instrument of 
data collection. The tools of the naturalistic evatuator are the 
unstructured In-depth Interview, detailed observation and content analysis 
of dociments and records. 
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The characteristic functions of the various evaluation tools of 
instruments have been described aid problems In their de>1giT~and adminis- 
tration have been listed. 

Things to do Qj Think About 

1. Develop a detailed list of facts, principle*, skills, and attitudes 
that you want your trainees to have learned by the end of your course. 

2. Have you been interviewed recently by soueone as part of an 
evaluation or a survey of some kind? Uhat *> you retneflfter Wat was good 
about the Interview? uhat did you find Irritating or unacceptable? Was 
the interviewer able to win your trust? 

3. Write an observation schedule on "Working Habits in the Office. 1 * 
Try tt on a colleague. Ask your colleague to then try it on you. 



\ 




CHAPTER XI 

PRODUCT EVALUATION OR PRE-TEST1NG OF 
INSTRUCTIONAL AND TRAINING MATERIALS 



Evaluation of Instructional and training materials Involves approaches 
and methods somewhat different from the evaluation of the effectiveness of 
development and training programs, A separate note on the Subject of 
"product evaluation" Is* therefore, In order. 

Earlier, In Chapter 1, we talked of two types of evaluation: 
formative evaluation and sumnatlve evaluation. The' purpose of formative 
eva luation Is to create evaluative Information during the Process of 
formulation of something* typically* a curriculum product - an Instructional 
aid, a lesson plan* or even a total curriculum for a training course or a 
school grade* The purpose of s uimatlve evaluation Is to sum up the effects 
of a curriculum or program or project and to discover what the Suirraatfve 
Impact has been and where. The Problems of Surrcnatlve evaluation, since 
they typically relate to Impact on learning behavior of Individuals and 
communities have been well-covered In other Parts of this monograph. In 
the following* we will focus en the formative evaluation of Instructional 
products, 

Prqdy^ct^Eva ljuatlon 

Product evaluation* also called Pre-testlng* has the objective of 
Improving a lesson plan, a set of posters* a book for new literates* a 
film for pregnant mothers* or a social studies curriculum for a school 
district before It Is finalized for release for more general uses. 
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The draft versions of a lesson plan* the typed manuscript of 4 book 
for new literates* the initial drawings of a set of posters* Cite rough-cut 
of a film are first used wi*th a small but representative yroup of potential 
readers and viewers. This gives the teacher or trainer an opportunity to 
test his or her assumptions, the readability level of language used, 
learner perceptions of the graphics* the organization of ideas and other 
related communicatlonal features of the material* The Information thus 
developed is used in revisions, redesigns and rewrites of the material as 
fol lows : 

« 

Version* Version, Version Release for 

Test, Test;, 9enera ' » se 

A development trainer may be Interested in the evaluation of any of 
the following instructional and training products; 
Leaflets, folders, handbills 
Li teracy primers 

Manuals* textbooks, followup reading materials 
Illustrations for use with tiie above printed materials 
Posters* charts, f iannelgraphs , flashcards 
Comics 

Games and simulations 

Audio programs of various kinds taped on cassettes 
Puupet plays 
Radio plays 

Films, video cassettes and TV. 
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Th e ProceU of Produ c t, Eval uation 

The process of product evaluation, or pre-testlng* Is about the same 
In case of all of the Instructional and training Products listed above: 
1. The evaluator should produce a dump, a replica* a rough print, or an 
approximation of the Instructional material to be pre- tested which Is as 
close to the final product as Possible. 

2- This rough correspondence to the final product should, again* be used 
with trainees In learning settings that approximate as closely to the real* 
life learning situations as possible. 

3, A sample of learners should then be exposed to the Instructional 
product arf* questioned about their perceptions, about what the product 
communicates to them, and whether or not they like using and learning from 
that particular Instructional Item, 

Khlle the steps Involved In Product evaluation or Pre- testing may 
generally be the same for all Instructional materials* the standards hy 
whldi different Instructional Items will be judged will be quite different. 
(See the section en standards of quality for judging Instructional materials 
In Chapter IX above. } 

Product evaluation may sometimes Involve more than one cycle of 
testing and revising. However, samples of learners used * n pre- testing 
of Instructional materials can be qutte small. 

Let us look at some examples of pre-testlng befow: 

Exagp le 1: Evaluating j j esson P^n 

Peter L, Klggs.' formerly of the Curriculum Development Center of the 

Veter L. Hlggs, "How to Evaluate Individual Lessons/ Lusaka, Zambia: 
Curriculum Development Center , Ministry of Education, Republic of Zambia* 
197a. 
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Governtaent of Zanfcia and now of Unesco* suggested thirteen steps In the 
formative evaluation of a lesson plan* (See next page*) 

Note step no* 6 that talks of administering pre* tests. Such pre- tests 
will, of course* nave to be designed In terns of the teaching- learning 
objectives of the lesson* Typically* these will be written tests* These 
tests will help establish, what^ Is called* the entry behavior level of a 
representative sample of learners. 

Observation of the lessonJStep 8) , Hote that while the administration 
of post-tests is planned in a subsequent step 9, persona) observation of the 
lesson Is Included fn the pre-testing process* Personal observation is 
Indeed a roost important step that must not be missed. This observation may 
be formalized using observation schedules; or It may be somewhat informal 
and impressionistic* Whatever the approach used* without such observation* 
it will be Impossible to know how the lesson actually "plays" In practice* 
How does the teacher handle the lesson? How do the students experience it? 
Are they able to relate the lesson to their everyday lives? What kinds of 1 
communication breakdowns seen to be occurring between the teacher and 
learners? What supplementation of the material might be necessary? is the 
material » perhaps* too long for one lesson and joust It* therefore*, be 
broken into Soulier units? 

Use of the Questionnaire in addition to a post- test . Note that In 
step 10. the application of a questionnaire is reconirwnded* This Is meant 
to generate that can be found neither through the post-test nor through 
observation of the classroom teaching* The ovaluator must a^k the teachers 
and ihe learners* as appropriate* about their inner experiences qf working 
with the materials* 
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STEPS IN THE EVALUATION OF INDIVIDUAL LESSONS 



p Prepare first draft of teaching materials 



I* Review Initial product 



Select trial ctassrooms 



Plan the evaluation 



Curriculum 
Specialists 



Meet with teachers involved In the pre- 
tests 



Administer (ire-tests 



Form program grouPs 



Observe lessons 



Administer Post- tests 



Administer questionnaires 



/ Curriculum 
Specialists In 
Collaboration 

q Hityi Teachers 
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Analyze results 
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^ Write 



final reports 



Revise materials 



174 



1 15 Curriculum 
u Specialists 
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fepurt_wr1tinQ - The writing of the report recommended In step 12 is 
aho important. A formal report not only creates a record* but also provides 
ttje evaluator with an opportunity to systematize his or her experiences; and 
draw deliberate conclusions and clear implications for the revision of 
teaching and tra1nfn9 materials. 

Example 2: Evaluating a set of posters 

1 Some of the ideas discussed i" the formative evaluation of a lesson 
plan under Example 1 above are directly applicable to the pre- testing of a 
set of posters. Some ottier ideas* to be discussed below* are unique to the 
evaluation of graphic materials. 

Objectives . He will nave to start with the question: uhat are the 
posters supposed to do? Are these posters teaching some Information? Are 
these posters teachin9 particular attitudes? Are these posters teaching 
some skills? What essentially Is lite "message content 1 * of the set of 
posters? 

Clientele . A series of questions will also have to be raised In 
relation to the clientele of those posters. Whom will these posters teach 
the new Information, the new attitudes, or the new skills? Are the 
clients literate or illiterate? Are they inales or females or are they 
mixed groups? Are they meters of particular clubs or participants of 
particular programs or are they members of the general prblic? This 
information will help us choose the pre -testing group* 

Setting , Questions may also have to be raised In regard to the 
setting in which the posters will be seen* will these posters be used In 
the classroom or will these be put on walls In the tojimunity center* or In 
the church or even in the town market? If thesewill be used in the 
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classroom setting* will these be Integrated Into lessons or will these be 
used as enrichment materials? The settlng(s) In which the posters wll) be 
used must be recreated during the pre-tests. 

Spe cimen materia ls. The posters to be evaluated will have to be 
prepared not fn the final form, but as close to the final form as possible* 
This applies to size, graPhic style* color* lettering! and even to the 
— quality of paper on which the posters wilt be ultimately printed Sometimes 
t more than one versions of the set of posters may have to be prepared: with 

and with* ut captions, line draping* versus photographs, etc. 

Pre-testlng . The first thing to do as part of the pre- testing will be 
to establish what the group of subjects chosen to look at the posters already 
fenow: Then the subjects should be given opportunities to view the posters 
under conditions very similar to those In which these posters will be 
later used or viewed. The e /aluator .should observe as the posters are 
being used* may have to give tests of knowledge to subjects; and may use 
questionnaires or Interviews to supplement test score data. . 

W hat to test posters for . A set of posters can be tested In terms of 
at least three types of characteristics: U) the Integrity and the 
continuity of the set of posters as a seti k {2) the graphic component of the 
posters; and (3) the verbal ^components of the posters. 

The first question to asfc In this c^se may be: Is the set, of posters 
really a set? do the posters go together? do they add up to a message 
with a particular Integrity? Do they have visual continuity? 

As part of testing the graphic con^onent* we will have to test the 
perceptions of viewers about the pictures, do they understand the 
pictures? do they find ftjpossfble to Identify with them In a cultural 
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sense? Do they like the style of graphics which nay be photographs, line 

/ 

drawings* cartoon figures, stylized, or realistic? Do they like the color 
coAfri nations? Questions about Vttenslty (ability to attract attention) 
may also be tested here. What parts of f a poster attract attention? What 
parts are missed or viewed only hurriedly? What parts are post remeiifoered 
when the posters have been removed? 

As part of testing the verbal component of the 'posters* *e will have 
to check readability of the captions. Are the captions understood? Are 
they appealing? i* the lettering used &f the right size? What do the 
captions add to the visual components? 
; Methods and samples , is it possible to conduct fairly satisfactory 
formative evaluations of posters (and other instructional materials) using 
small samples of representative users. The writers and designers of sucli 
materials must* however* be themselves Involved in these evaluations; they 
should themselves conduct the interviews and make personal observations. 

Two methods have typically been used In pre- testing of Instructional 
materials - the "tutorial method 4 * and the "group-based method^. 1 * in the 
tutorial method a single learner Is involved at a time. Oral and written 

verbal behavior as well as non-verbal behavior of the slnglejlearner form 
m " i 
■the basis for revision and a rapid and subjective data analysis Is under- 
taken. In the group-based method a representative sample ofj the target 
population is used In a group situation* a description of the aggregated 
group behavior forms the basis for revision and systematic statistical 

i 

analyse* of-data are often undertaken, Both *nethods have. been found to 
be equally effective* indeed* no statistical differences have been fouiuJ 
between the effect J venessNif Uiese two metiiods, 
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Evaluation of Instructional materials (also called pre- testing of 

if 

Instructional materials, or simply product evaluation) presents a set of 
problems somewhat different from those faced In the evaluation of the 
tnpact of programs on the behavior of Individuals and communities* the 
basic steps In the process of pre-testlng of Instructional materials are 
given and the need for a set of standards to judge quality of materials Is 
Indicated* Two examples are provided: one Involving the pre- testing of* a 
lesson plan* and another involving the pre-testlng of a set of posters-*. 

Things to Do or Think About 

~ ■ — j 

1* Adapt the "Steps In tie Evaluation of Individual Lessons" given In 
A 

this chapter to the evaluation of one of your own lessons* 
2* Using the Ideas Included In this chapter (and particularly In 
Examples 1 and Z)» can you develop a scheme for pre-testlng an educational 
radio broadcast? 



CHAPTER XII , 

OESlGWHG EVALUATION STUDIES* AKO 

THE CONCEPT iCF DESIGN la HATUHALISTIC EVALUATION „ 

In the dictionary meanings of the termi to design is to develop a 

conception of something or ts to prepare preliminary plans or sketches for 

something* In this sense of the word design* all evaluation studies must 
p 

have a design. We Must have a conception of what we want ''to do» why* and 
we must make some preliminary plans about how to go about doing what we want 
to do. * 

\w the literature of research and evaluation* however* design has a 
highly technical meaning. To have an evaluation design is to follon a set 
of procedures that will increase tne internal and external validity of 



equation results (when using the scientific paradigm), or tne credibility 
and fittfngness of results (when using the naturalistic paradigm). 

In the following, we wifl discuss rather briefly the problems of 
evaluation design in both the scientific and the naturalistic paradigms 

Design \\\ the Scientific Paradj gm 

Within the scientific paradigm, design typically means "experimental 
design". There ha±> to be a sampling plan, and random samples nust be 
obtained. Evaluation variables must te controlled through various mechanisms. 
Treatments should be well-defined and applied selectively to chosen samples, 
instruments are often structured and statistical techniques are applied to 
the analysis of data collected. 

It is beginning to ne widely understood* however* that 'true 1 experi- 
mental designs are seldom possible in education and development. Random 
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samples do not always make sense when dealing with special categories or 
subjects* In particular community contexts. Control of variables and 
treatmer*s Is often Impossible* Evaluators are* therefore* now being 
offered "quasi -exper f mental designs' 4 - evaluation designs that are half- 
way expert-rental. When using quasi-experimental designs* we try random 
assignment of treatments, if possible; but control when the data wilt be 
collected and lr°?,Wtal- 

s 1 qr? \n the Hajturajlstlc Parajjqw 1 

i 

Within the naturalistic paradigm* they t&lk of aexperlmcntal (non- 
experimental) designs. Ihey do not seek to experiment upon Individuals* 
groups and communities. T ;»t naturalistic paradigm demands that human 
behavior be studied as It nccurs naturally, within tts natural socio- 
cultural context. Ihls means that neither the .Intervention of treatments* 
nor the control of variables and other contingencies Is sought. 

Thus, the design In the naturalistic paradigm has to be, what Cuba and 
Lincoln have called, an ef^rg v: «t design .or a rolling design. Plans and 
procedures for the evaluation stucfr have to emerge within the realities 
of the Held to meet needs as they are seen and to follow leads as they 
present themselves. Instrumentation has to be responsive to the data that 
happens to become available. Samples often get nominated and become 
exhausted when there fs satiation of data and regularises In responses 
hfiOln to show up, 

ReljabnUy^a nJ Validity 

fiesea :he-s and evaluators working within the scientific Paradigm 
swear by reliability and validity. 
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Reliability applies to a test or another measuring instrument. It is 
tie fine J as a reasonable consistency In results' obtained In a sequence or 



group of repeated tests ana measures, A reliable test Is one which gives 
consistent results in different* applications to the same subject within a 
reasonable tine-frame. Or* performs consistently when used by different 
e valuators, with different subjects. Reliability Is necessary though nut 
sufficient for validity. 

Validity is the extent to which a test measures the thing it is 
supposed'to measure. Support for validity may be logical or empirical. 
The test 'tens may have been properly derived from accepted premises by 
rules of logic* or assumptions may have been based on supportably empirical 
evidence. 

Inter nal and Extendi Validity 

The concept of validity applies not only to tesis and Instruments 
but also relates to the more geneYal concerns of evaluation design. The 
results of an evaluation study and the conclusions drawn from these results 
must be seen as warranted* convincing and acceptable - that 1±>, they must 
be seen as valid, 

tet us look at some of the assertions that ©valuators could make on 
the basis of their studies, and at the possible objections that could be 
raised to the validity of Such assertions: 



Assertion by eydlualur fr 




The trainee group has shown 
considerable learning as evi- 
denced by the high level of 



Maybe this yroup W as familiar with 
the content of the training course 
even before joining the course* 



performance on the final test 



Maybe tike test was easy or the 
grades have been inflated. 
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Acfult attitudes towards literacy 
hfve changed drastically 
because of the project. 



The group of farmers who under- 
took leadership training at 
the training institute had 
assumed actual leadership 
roles In the coimun f*v taore 
often than these farmer^ who 
did not join leadership 
training. 



The farmers 1 training course 
Increased the overall 
productivity of farmers who 
attended by 15% In a year. 



Hie Introduction of the role of 
the Family Health Education Worker 
has changed the level of health 
In the selected communities from 
'Poor" to "Medium" 



Maybe they have changed not because 
of the project* but because of the 
President's speech on national radfo, 

Kaybe they have changed not because 
of the project* but because of 
new *y- opened textile factory has 
declared Its preference r or literate 
4<d set" -literate tabor. 

Mavfo the farmers who undertook 
icadi *ship training were already * n 
1ead<?»V^P positions and wanted to 
Increase thel r effectiveness as 
leaders t 

Kaybe the fanners who joined leader- 
ship training were a self-selected 
group* ffred with the ambftlon to 
rapture the new leadership positions 
opening up In their communities. 

Maybe the other group of farmers that 
is not doing well* Is different from 
the successful leadership group tn 
Important socioeconomic characteris- 
tics, 

Maybe the productivity Increase for 
these farmers last year was 203L 

Maybe similar farmer groups 
elsewhere have shown similar Increases 
In productivity levels. 

Maybe this Is because of the heat and 
drought of the last year that killed 
all mosquitoes; and the famine 
relief high protein food aid that 
was provided to families In the area. 



These are some examples of the assertions that could be made and the 
Challenges to their validity, Professors Donald T. Campbell and Julian 
C> Stanley' have listed twelve different threats to the Internal and 

1 Donald T. Campbell and Julian C. Stanley* Experimental and Quasi- 
Expe j jmenta? fiesjgus ^ for jtes ea rch. Chicago, 1117: EanTTkfiaTiys 1963- 
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external validity of evaluation studies, ^valuators should find their list 
roost Instructive: 

(A) Internal validity 

h History , An outside historical event* such as a presidential 
speech, or the enthusiasm generated by a newly announced economic plan 
could challenge the validity of the evaluator's claims, 

Zh Maturation , Individuals being tested as part of the evaluation 
may mature and grow in such significant ways that they may behave tike 
different people by the time an evaluation study is completed, 

3, Testing, The first test may teach the Items on the test and other 
re la tea and implied information. The same test (or an equivalent second 
test) may not then measure real changes brought about by the program, 

4h instrumentation . There may have been no changes In the reality 
but only in the calibration of instruments studying that reality,^ 0r» 
different observers and examiners may have given different scores for the 
same unchanged reality. 

5 kt*t >tkdj regression . This Is a statistical phenomenon. 
Extremely high or extremely low scores on a first test tend to move towards 
the mean of total scores during a second test, Thus> changes In the 
scores on a second test may really have nothing to do with respondent 
groups* program methods ♦ or program effects. Statistical regression 
occurs specially In cases where group* have beui selected on the basis of 
extreme scores , 

6, Selection , Biases In the selection of learners for training, 
Interviewing and testing my threaten the validity of results, 

7, Experi me ntal mortality. Those Initially covered by an evaluation 
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study may cease to be Participants In the evaluation. They may drop out of 
the program „r may move away In search of food or work, 1hus t the residual 
group may no more be representative of the group or contramlty being studied. 

Select Ion- maturation Interaction . The peculiar chemistry of the 
selection process of subjects In an evafuatlon study and their maturation 
together may show effects Independently of the program fnputs and processes, 

(B) External validi ty 

9, The reactive and Interact We .effect of testing . The Pre-test may 
Increase or decrease the sensitivity or responsiveness of the respondent to 
certain program treatments aPPlled as part of the evaluation, 

10. S election-treatm en t Interactions , Th* pecutfar chemistry of 
selection of respondents and the Instructional and organ Izatfonal treatments 
may create effects that falsify results regarding real program effects, 

11. Reactive effects of experimental arrangemen ts. Persons and groups 
show one set of effects of a treatment within the experimental setting* but 
not In non-experimental, real-life settings. Or, In some cases* experi- 
mental conditions may be much too artificial. 

12. HtfUlple-treatment Interferenc e. When the same grouP Is frequently 
tested* or Interviewed many times In different connections* results may 
become confused. Effects of a test and an Interview cannot be erased from 
the minds of respondents and the first test or Interview may Influence 
later testing and Interviewing In ways that we do not understand. 

The Purpose of evaluation design Is to reduce the above mentioned 
threats to the validity of evaluation result*. 



m 

Credibility and FUt fitness 
I n naturalistic Evaluation 

In Chapter VIU (Models of Evaluation), we have reproduced a table 
developed by Cuba and Lincoln comparing scientific and naturalistic terms 

it 

appropriate to various aspects of rigor in research and evaluation. They 

pair them as follows: * 

Sclent .He term Naturalistic term % 

Internal Validity Credibility 

External Validity/ - fittlngness 

General liability 

\ * 
Reliability Audltabflfty 

Objectivity Conff ratability 

Qbjecti vitV/Conf irmabi lity , Too often naturalistic inquiry is 

dismissed « subjective while scientific inquiry is considered objective . 

This is quite absurd because both types of inquiry (scientific and 

naturalistic) work with data provided by individual subjects, It is* 

therefore* always "subjective". 

The "subjective"* however* becomes "objective** when ft is confirmed by 

a number of subjects or judges. Indeed* the significant word here is 

conffnnabf ltty. If naturalistic data can be confirmed by others, then it is 

objective. 

Reliability/ Audi tabll Ity . In naturalistic inquiry ne can not talk uf 
reliability because we are nut dealing with structured instruments that are 
supposed to per f una consistently. In naturalistic inquiry* the evaluator 
oecomes the instrument. Instead of reliability* Guba and Lincoln -have 
proposed the concept uf auditabtlity. AudftaLility weans simply that ti^ 
work of one evaluator can be tested fur cunsisteni.y by a second evaluatur* 
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eternal Validity/Credibility . Internal validity gets translated Into 
credibility In naturalistic Inquiry, The evaluator can Increase the 
probability of produclnq^credlble findings by allowing sufficient time for 



respondents become familiar with the evaluator and the evaluator can check 
on own preconceptions; by establishing a proper rapport but by avoiding 
deep personal Involvement; by checking on biases of buth respondents and 
of self; by checking data for Internal consistency through structural 
corroboration (each piece should validate the other and all Pieces of data 
should add up to a plausible whole); and, finally, taking data back to 
respondents for their check, - 

External Validity (General1zab1Hty ) /ritt1ngness , The concept of 
c generallzablllty Is Itself undergoing change and some researchers are 
questioning the very Idea of context-free generalizations. In naturalistic 
evaluation the concept of gftnerallzabfllty (external validity) gets 
translated Into the concept of flttlngness* The evaluation findings from 
a naturalistic stuoV should become the working hypotheses that would fit In 
similar programs It* similar contexts elsewhere. •> 

b 

So me Ideas on Sampling 

Validity and general rlgurousness of evaluation studies can be 
Increased by following proper sampling and design methods, We begin by 
presenting some simple Ideas on sampling. 

A sa mple Is a portion, part or piece taken or shown as a representative 
of the whole. Sampling Is often a practical need, ^valuators may deal with 
programs with brood scoPei covering hundreds of thousands of people, Jhey 
can not go to each and every merter of their populations and ask them the 



the novelty of his or her entry In the field to wear off so that 
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questions for*whfch they want answers. Instead they want to select a snail 
number of respondents In such a manner that tfie sample Is representative and 
can be studied to make inferences about the whole. 

We should explain the two woros population and representativeness used 
In the paragraph above. Jn everyday meaning of the term* population covers 
all the people - men, women and children* young and old, farmers, workers 
and housewives - living In a particular community or nation. For the 
evaluator* population is the total group of people In which the e valuator 
Is interested. It may be all women in a country in the child-bearing age* 
all people suffering from lung diseases, all textile workers Or all new 
literates in a region or a township. Samples are drawn from such populations. 

Sam, s have to be representativ e* that Is, as parts they have to 
represent the whole from which they are drawn. 

There, have" been many advances In sampling theory. Statisticians have 
worked out formulas whereby they can tebt the representativeness 01 their 
samples and calculate the probabilities of error. 

Size is an Important consideration In selecting samples. Clearly the 
perfectly representative sample of a population Is the population itself. 
Generally speaking, the larger the sample, the more representative It will 
be of the population. But unnecessarily large samples will not be good 
samples, We have to have the right size of sample that' Is both economical 
and representative. 

On the next pag?« we have reproduced a tabje that can be used for' 
determining sample sizes for various population sizes. Let us also look at 
some frequently used types of samples. 

On t 
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" TABLE FOR DETERMINING SAMPLE SIZE FROM A GIVEN POPULATION 

9 



N 


S 


N 


s 


N 


s 


10 


10 


220 


140 


* 1,200 


291 


J5 


14 


230 


144 


1,300 


297 


20 1 


* 19 


" 240 


148 


1,400 


302 


25 


** 24 


250 


152 


lvSOO 


306 


30 


28 


260 


155 


1,600 


310 


35 


32 


270 


159 


1,700 


313 


40 


36 


280 


T62 


1,800 


317 


45 


40 


290 


165 


1,900 


1?0 


50 


44 


300 


169 


2,000 


322 


55 * 


48 


320 


T75 


2,20$ 


32/ 


60 


52 


m 


T81 


2 t 400 


331 


$5 


56 


360 


186 


j 2,600 


335 


70 


59 


380 


191 


2»800 


338 


75 


63 


400 < 


196 


3,OQ 


341 


80 


66 


420 


20T 


3,500 


346 


85 


70 


440 


205 


4*000 < 


351 


90 


73 


460 


210 


4*500 


354 


95 


76 


460 


214 


5,000 


357 


1(K 


eo 


500 


217 


6,000 


361 


110 


86 


550 


226 


7,000 


364 




tit 


600 


CJH 






13? 


97 


650 


242 


9,000 


368 


140 


103 


700 


248 


10*000 


370 


150 


108 


750 


254 


15,000 


375 


160 


T33 


flOO 


260 


20*000 


377 


-170 


118 


050 


265 


30,000 


, 379 


180 


123 


900 


269 


40 t 000 


380 


190 


127 


950 


274 


50,000 


381 


ZOO 


132 


1,000 


273 


75,000 


382 


21 0 


136 


MOO 


205 


100*000 


384 


Hotet 


H Is Population size* 
S Is sample size 









R. V. Krsjcle and £>- Morgan, "Determining Sample Sl2e for Research 
Activities," Educational and Psychological Measurement , 30:607-610, 1970.' 
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Handoui sampling ,^ A random sample results when selections are made 
purely on the basis of chance* without any underlying system or pattern, 

s 

and when each item or person In the population being studied has/had an 

equal chance of being Included in the sample* Random samples of appropriate 

o 

size are most likely to represent all the characteristics and exact 
distribution of the t&tal population of evaluator's Interest* One method 



of taking random samples Is to arrange the population In some way, assign 
numbers to it, and then draw some nu&bers randomly* Wheref the populations 
"are big and the numbers to draw /row are large, prioted /ables of random 
numbers can be used. / 
+ Random sampling may often be applied sequentially in evaluation studies. 
Geographical regions of a country may be selected randomly, followed 
sequentically first by the random select top of communities within the 
randomly selected.reglons, and then by the random selection of adults in 
the randomly selected cotfnunl ties, Again, randomly selected adults could be 
assigned to different learner groups through subsequent random selection. 

List sampling . List sampling Is a mod ff lea t ion of the random selection 
method. The population of Interest to the evaluator Is arranged In a list 
according to some rule - alphabetically, for example - and then every nth 
number Is* selected from the list. Tor example, every Sth or every 20th 
number may be picked up depend ingmpon the size of the population and the 
size of the sample being selected. The starting point in the selection 
process can be randomly siflecTed to meet tike criterion of equaj chance of 
selection for each urihA *>^* 



ch un 
n£* 



Area sampHng * In area sampling, some geographical locations may be 
randomly selected from all available sites am* then all appropriate units 

9 / 1 Q 1 " 

ur/ 15J 
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within the selected arenas may be studied. ^ 

Stratified samplin g. The population of Interest to in evaluator may 
be divided Into distinct socio-economic strata. 0r t the population may be 
stratified according to age groups - children, young* middle-aged and very 



old. It? such ca^es, stratified sampling may be used. ^accordance wfth 
proportions In ih& total population^ samples may be drawn proportionately 
and randomly ,from each of the population strata. / 

Purposive (theoretical or elite) sampling^ The naturalistic evaluator 
or researcher may often need not a random sample fcut a purposive sample, a 
sample that; fulfills his or heir theoretically determined needs. The 

s 

evaluator may be interested ijot In any randomly selected group of adults 
)n a commfjn1ty t blit In two or'three people who are supposed te serve as 
the community's gate- keepers. The evaluator may tje Interested* that Is* 
In 'sma^l elite samples* 

Some Simple Designs for Evaluator^ 

A few designs Qf Jutejcasl to evaluators are presented below. These / 
descriptions are based V 1 the work of Campbell and Stanley referred to 
earlier. \" 

(1) The One-Shot Case Stud y 

Campbell and Stanley call it a pre-experlmental design* There Is a 
total absence of control. A program treatment (X) Is followed by observation 
(0): ' > . ' ' / 



While a case study Illicitly compares Us results with similar events 
casually observed or read and remembered, the case study can be strengthened 
by more systematic comparisons* At least one more comparison should be 
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attested* We ^fiotild remember that this so-called pre-experimental design * 
Is the essential tool of the naturalistic evaluator* 
The 0ne-6rojiP_Ere test-Post test Design 
This is also considered a pre -experimental design and can be represented 
as follows: 

o 1 X o 2 

A first observation or pretest (0,) is followed by program treatment 
(*)* after which a second observation or posttest (0^) is recorded* ' 

^valuators will be using this design often in their evaluation studies* 
They should* however* do their best in del ndlng their resuHs against threats 
to their validity; or In qualifying their conclusions tn the light of 
effects of history* saturation, testing or instrumentation as discussed 
above* (We have earlier discussed twelve threats to tye internal and 
external validity of evaluation results* It will be agood idea for 
evaluators to develop the habit of checking their results in regard to each 
Qf these twelve threats* every tine they design or complete an evaluation 
study*) 

(Hi) The Static-GrouP Comparison 
^ It is a design in which a group which has been subjected to a program 



treatment Is compared to another that has not been; 



"1 



I 



This again Is a design under many threats of validity* The most 
obvious ones are that of selection (the two groups may have been different 
to begin with) and mortality (subjects in the experimental group or the 
comparative group may have left the groups for some reason.) 
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(U) The Prrtesff-Fosttest Control Group Design 

r 0 
Campbell and Stanley call It a "true* experimental design, Ttrq, 

. samples are randomly selected from the same population. One. Is assigned 

a program treatment and the other Is not: 

This deslgft meets most of the standards of Internal^vatldlty quite 4 

1 ' ' / 

adequately* though care must be taken fn generalization of results tp the 

general Population, j 

(vj The Posttest-O nl y Control Group Design , - * 

This Is another example of the true experimental design* The pretest 

* suggested In the design Immediately preceding may not always be possible. 

It is not even necessa^* If randomization In group selection can^be 

assured.* The design then takes the form: / 

R X ! 0, 

R / ° 2 I 

(vl) The Quasi-Experimental Designs; Ihe .Tfme-Serles Experiments 

The M me series design Involves Periodic measurement ojr some Individual 

* \ 
or group boy»J>efore and after the Introduction of some Program treatment 

and the study of the "discontinuity" Introduced In the pattern of betavtor 

In time: ' 

f ' °1 ■ °2 * °3 °r' X °5 °6 *°7 ,°8 

,e e^aluator using this design must specify" In advance the expected 



time relationships between the Introduction of a Program treatment and the 
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manifestation of .Its Impact. The relative isolation of the group from 
outside Influence should be ensured as well as some consistency In the 
conditions. 

The above design can be strengthened by working with two groups In a 
time series as follows: 

0 0 0 0 X 0 0 0 0 
0000 0000 

(vil) T^e Honequlvalent Control Group Pesi gji 

This is a design in widespread use because it fits the realities of 
the world of education and development as often faced. Too often evaluators 
have to work with already formed groups and classes and can not assign 
meters to them randomly. 

Thus* the design takes the form: 

0X0 
* 0 ' 0 

We should note the similarities between this quasi -experimental design 
- nd the "Pretest-Posttes t Control Group Design* 1 which was described above as 
a true experimental design. The essential difference between the two 
designs Is that in the case of the "Pretest-Posttest Control Group Design" 
the treatment and the control group are chosen randomly while in the 
"Nonequivalent Control Group Design* 1 discussed here, the groups are not 
randomly chosen and hence are nonequivalenL 

Summary 

The problems of reliability and validity (both Internal and external! 
are discussed 1" relation to the scientific as* well as the naturalistic 

» 193 



192 

paradigms. Twelve different challenges to Internal and external validity 
as suggested by Campbell and Stanley are listed. There are no perfect 
designs that will take care of all the threats to the validity of evaluation 
results. However, appropriate sampling methods and choice of designs will 
enable tralners-evaluators to systematically consider rival explanations and 
to quality the results they *b obtain. 

7h .qs to to or Think flboul i 
1. Examine the conclusions of any evaluation study recently done by a 
colleague In your training Institute or In some other development setting. 
What are some Possible rival hypotheses or explanations for the assertions 
made by the evaluarors? 

Z* Look at the table of "Assertions by evaluators - Objections to 
validity* 1 Included In the beginning of this chapter. What kinds of designs 
could have been used In each case to defend the validity of conclusion' 
arrived at by educators? 
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CHAPTER XI H 

I HPLEKEHTi KQ EVALUATION STUOIES: 

PROBLEMS Of DATA COLLECTION 1H THE REAL WORLO 



Rany of the problems 0 f liaple&entlng evaluation studies have been 
referred to directly or Indirectly In other parts of this monograph* A 
systematic and self-contained discussion of the practical problems of 
conducting 1 evaluations say* however* be more helpful and is Included below. 

Circumstances are sometimes stronger than men are. An evaluator can 
not control wind and weather, nor drought and famine. One can only cope 
with such circumstances and do the best possible. But many other possible 
sets of circumstances can be anticipated and one should be ready for them. 
/ 

A Hew Set of CpjjeQja] Relationships 

Evaluation Is business unusual* Even when It Is an evaluation of your 
own work by yourself* you pe tjrb the existing relationships with your 
colleagues* It Is Irop^tant that you keep your feeling of self-Importance 
In check and Inform all concerned about what you are doing and why. Personal 
fears must be assuaged and professional jealousies must be relieved. 

Evaluation wilt always make jujjusual demands on those who work with you 
in the office and In the field. The evaluator has to transform ids 
officers* colleagues and assistants all Into professional collaborators. 
The evaluator has to receive the blessings of those above* establish fair 
exchanges with those at the same level* and receive help from those below* 
not by ordering around but by sharing excitement as veil as .credit for the 
work done* Que acknowledgment must be made, ijoth verbally and in writing* 
to those who provided. advice or assistance. 
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Training of Field investigators 

In most cases, you as equator will not be able to collect alt data 
single-handedly- Yon will need the assistance of collcagurs and other 
field workers In the collection of data. It Is Important that those who 
have been mobilized as field Investigators are provided appropriate training 
and orientation. The evaluator way not always want to Inform the Held 
Investigators about the evaluation hypotheses or questions to keep out the 
Personal biases of the field Investigators. But the field Investigators 
must be fully trained In the requirements of administering the evaluation 
Instruments. Such orientation and training may have to be fairly extensive 
If In-depth Interviewing Is Involved. 

U Is Important that the evaluator Is able to stay In constant touch 
with the field Investigators to be able to answer their questions and solve 
unanticipated Problems. 

Piggyb acking on Existing 
Instl tutl op al Resources 

\ 

tt Is Important that tralners-evaluators learn to piggyback on 
existing Institutional resources. This Is especially Important In the case 
of transportation facilities. Travel arrangement to the field should be 
made to fit the travel Plan* of various officers from the parent department 
as well as other sfster development departments*' 

pe aHng ylth the Jespqndent s * 

The evaluator can not anticipate famines and Funerals, but must be 
aware of the seasons for migration of potential respondents * thetr dally 
Patterns of work, and their festivals and hot f day. * 
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The Investigator must be able to stay In the area long enough to wear 
off the novelty effect of his or her being there* to establish a rapport 
with the people* and to administer the questionnaires or to conduct the 
Interviews. , 

The evaluator may have to use a third person to accompany him or her 
to conduct interviews with young mothers who may feel embarrassed being 
^all alone with the Investigator. In such cases, the third person will have 
to be chosen with care and the rules of conduct during the interviewing or 
questioning will have to be properly explained. 

There will be situations when respondents will expect to be pa 1^ for 
being subjects of an evaluation study. As we have indicated elsewhere, 
evaluators (and researchers) should not pay for data unless 4 respondent 
will be losing wages In cash by participating in the evaluation study. 

Changes In Samples and Instruments 

In naturalistic evaluations, sampling Is purposive and samples are 
developed and redefined to suit the circumstances, in the so-called 
scientific evaluation, samples are pre-detennined and pre-selected. it 
will often happen that the evaluator Is not able to collect data from the 
pre-selected sample and Is obliged to make substitutions for the respondents 
lost or Is forced to make-do with smaller samples. * 

It Is not posslble.wlthln the scope of this short chapter to deal with 
the complex Issues of sample attrmgp and sample substitution. A general 
piece of advice can be offere^T^fowever. The advice Is tfiat evaluators must 
keep 3 precise and honest record of the changes made In the samples so that 
appropriate judgments can be made at the stage of interpreting data and 
results. 
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There trill also Be Instances when changes 3n the evaluation Instru*, 
ments wilt be necessary, Sotne questions may not be understood by the 
respondents In an evaluation study, Some questions may be unanswerable* 
and some others the respondents may refuse to answer* The e valuator should 
be In touch with the field Investigators (where field Investigators are 
Involved) to discuss problems and nuke the necessary changes* Changes made 
In the Instruments should be followed unlformally by alt field Investigators. 
Clearly* such situations can be avoided by proPer pre-testfng of evaluation 
inst rumen tJ* 

Handling of Completed Instruments 

Problem; can arise from careless handling of completed Instruments. 
Questionnaires and Interview schedules can get tost* Data Is precious and 
should be treated as such* Field Investigators should be Instructed ' 
clearly In regard to mailing and despatch of data. Should H be sent by 
hand with officials travelling from the field to the city office? Should rf 
It always be mailed? How should It be packed? Should It be sent by 
registered mall? 

Completed questionnaires and Instruments can get mixed up In the 
equator's office, These should be properly marked and coded as Son as 
received. 

Summary 

Various problems can arise In the course of actual Implementation of 
on evaluation stooV* Colleagues may not collaborate?* Field Investigators 
may mishandle the administration of evaluation Instruments* Getting to 
the field may be difficult. Respondents may be Inaccessible and Instruments 
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need last-ninute changes, finally, collected data nay be lost through 
careless handling. White an evaluation study may never be completely 
problem-free* the evaluator should be able to anticipate some of these 
problems and fix others as they appear, 

Things to Do or Think About 

U What are 1 some of the problems that you anticipate i n the course of 
data collection In your setting? 

2. What are your suggestlnns for valuators in regard to establishing 
fruitful collaborative relationships with their colleagues and subordinates? 
3* Can you think of erases where problems In data collection in the field 
killed an evaluation study? 
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chapter xiv 

THE PROCESS AND lECHHlQUES OF 
DATA ANALYSIS 

After tests and questionnaires have been administered* fntervfews and 
field observations have been conducted* and relevant records and documents 
have been collected* what we then have Is raw dat a. Raw data* fn Itself* Is 
not Information . Information has to be systematical!/ produced from avall- 
r able ran data for use fn Programming decisions* 

The Meaninl of Data Analysis 

The process of converting the raw data fnto evaluative Information Is 
called data analysis. The skills and techniques of data analysis Involve 
coding and scoring* combining and weighting, standardizing* ranking* 
tabulating* making graphs* developing crossbreaks and running stat1stt£3J 
tests for correlations and differences. Or* the process of data analysis may 
require developing case studies and profiles; thematic analyses and writing 
analytical reviews or product evaluations of currlcular and training materials. 
We will deal briefly with most of these approaches and techniques In this 
chapter. Statistical' analysis of data ul 11 be discussed separately In 
Chapter XV. 

Some Tools of Bata Analy ses ^ 
The following are some of the tools of data analysis: 
a 1. A good supply of ruled and plain paper 

' 2. A supply of lead pencils and a pencil sharpner 
3* Erasers 
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4. If possible* a bottle of liquid paper (or another correction fluid) 

5. A pair of sclsscrs 

6. Scotch tape with d1s*.?nser ' i - 
7* Paperclips and pins* and 

6, A set of colored pencils. , , 

A Basic Suggestion ' 

u - * 

0 

the process of data collation and analysis* *£lte only on one side of 

the paper. Use t separate sheet of paper for each single Idea ana* theme that - 

^ % * 

you develop. This will help you later In trying different organizations of 

* ■ < "* ' 

the oatertal. You do not have to use nice and fresh paper fordftis stage of 

data processing. You should use discards froa cyclostyled materials and any 

other scrap paper you can get hold of. for raking tables by hand, use ruled 

paper so that rows of data can be read without confusion. Be careful about 

the spacing of niwfcers In columns; 

125 125 
5 5 
11 Is correct . 11 Is not correct 

Do not write over your own writing. Use * n eraser; or strike *out and 

write afresh. 

f 

Clustering and Identification of Data Pieces 

For the sake of convenience* let us call the tests, Interview schedules*- 
observation schedules* and questionnaires, filled ant) returned by Investigators 
and respondents, data pieces . The very first thing to do when all the data 
pieces are In, vtill be to arrange and Identify the various pieces by numbering, 
thev* Different arrangements will be appropriate In different cases* Where 
respondents are notuanohymmi pledes may be arranged alphabetically. Other 
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arrangement may reflect clusters of data pieces; and these data Pieces may 

be clustered by sex* age* religion or ethnicity; by training course, batch 

or year_; by region* province or district; by literacy Jeacher In charge; In 

* f 

terms of trained versus untrained groups; and by training methodology used. ' 

An example of data pieces organized by clusters my look J ike Uie 
following: 

Region 'X 

Primer 1 * - 

1 Male 1 X 

2 Hale 2 ¥ 

3 Hale 3 

4 Ferrule 1 
, B Female 2 

j 

Primer 11 < 
G Male 1 

7 Hal4 2 

8 * Female 1 
g Female 2 

10 Female 3 

Region Y , 

Primer 1 * 

11 Kale I 

12 Hale 2 

13 Kale 3 • * 

14 Female 1 * 

15 Female 2 

Primer II 

16 Kale 1* 

17 Hale 2 

18 Female 3, 

19 Female 4 
2C Female 5 

a * 
Such clustered organization and Identification of data pieces helps at 

* 

the later stages of data analysis. 

Once organized according to need* all data Pieces should be given 
permanent numbers In the upper right -hand comer on the face of each piece, 
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using a color code* v * 

Jf a whole seUof Instruments — an achievement testi an Interview 
.schedule* an observation sche&le and a questionnaire — have alt been used 
with the same one 9rouP of respondents^ then a matching numbering system 
should be used. For example: 



Hang > 
* * 


Interview 
Score U) 


Observation 
Score (0) 


questionnaire Test 
;core (i)) Score (T) 


Abrfc 


* 

l-l : 




<H 




Bint* ^ 


1-2 






T-2 


Camaro 


1-3 


0-3 


Q-3 


T-3 


Daudt 


1-4 


0-4 




T-4 


Elice 




0-5 




" T-5 


Fdkcuri V 


1-6 

A 


,0-6 


Q-6 


T-6 " 



Hake sure that you wrfte f and T and 0 and Q clearly enough so that 
one U not confused with the other. Note that In the above display* 
* Daudl's questionnaire ts^jnlsslng, as^is £>lce*s Interview schedule. 1 How- 

, ever, Fakourt still gets numbers 1-6, 0-6, Q-6 and T-6 for his data pieces. 

^ ' v V 

In other words, all data pieces for [he same one personjaet matching numbers. 
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The Heed for Immersion In the Data 



After the data pieces have been arranged and numbered, It will be time 

Y > 

to do two furtner things; to recollect the evaluation questions that needed 
to be answered by the. evaluation study* and to get teinersed In the data 
already collected. ' *\ * 

Write ut the list of questions you want' the data to answer. If there 
* are sub-questions to the questions* write them out'also. For exalte, an. 



evaluator may write his or her questions as follows: 



■ ■ fr 

er|c 
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» ? * : 

1. How are trained assistant aduljt education officers different In regard 
to their overall performance from untrained /assistant acful t education » 
officers? 7"^ . t 

* 4 ^ 
t 1.1 How do they differ In regard to their technical knowledge about 
development and adult education? 

1.2 How do they differ In regard to their knowledge of the literacy 
methodology being used In the prog rami 

1.3 How do they differ In regard to their supervision styles* and 
diagnostic and problem- so Wing sfctlls? 

_ 1.4 ilcw do they dl rAr 1i terms of their attltudlnal orientation to 
*r adult learners', rural communities and theljr own worfc? 

Armed with Such a list of questions, It fs time to begin the Inversion 
In the data. By Immersion In the data* we mean going carefully through all 
the pieces of data, piece by piece* page by page, Item by 1 teen* studying all 
the responses* and making careful ^ written notes. You should take note of 
the expected* of the unexpected and\of the curious; of the emergent pattern 
and of the seeming relationship, as ^qu go through the dat£ This Immersion 
may require more than one dip* that 'Is* you may have to do more than one 
reading of the data pieces. The time useo In going through this process 
Is always well-spggt. Therefore, be -patient* 

Possibilities and limitations of the data collected . This will alto be 

the time to tflscover.the unanticipated possibilities of the data. For 

example* a questionnaire used wfth students of agricultural extension to 

evaluate their attachment experience* may be full of Information about 

prevalent practices on butchering meat an1mals*gj^on the popularity of 

poultry farming* In a particular region. On the other hand, serious problems 

» 

;nay be discovered with the dS?a during the Inversion process. Some 'questions 
fn^he test may have been consistently misunderstood. Some other questions 
may have received "socially acceptable" responses* not the real answers* 
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Song pieces may have to be discarded altogether for being Incomplete or dis- 
honest. H may become clear to the evaluator that available data villi not 
wake an overwhelming case for or against a particular position or approach; 
and the evaluator may have to warn readers against drawing unwarranted 
conclusions. All this should be taken note of during the process of 
Immersion^ In the data. 



Developing Hotels of Oata Analysis \ 

In the process of data analysis, we have to work back and fo\h from 
both ends — from the direction of the raw data we have and from the side of 
the final Information we need to produce. 

To make sure that we do not get lost in the details of the data 
.collected* we must keep In mind the questions mat will ultimately have to 
be answered. He jojst also develop a model or plan of data analysis. This 
so-called model of data analysis* may be nothing jocfre than a reincarnation 
(rebirth) of the evaluation design used earlier In conducting the evaluation 

In Chapter XII » we had listed some of the typical designs used by 
evaluators: simple comparisons between groups* before and after designs, 
etc. Some evaluation studies may not fit, the experimental or quasi- 
experimental oodes discussed In Chapter XII. Some evaluators* for instance/ 
may be Interested In the study of patterns of perceptions and configurations 
of relationships in the field. Given below are some examples of models of 
data analysis actually proposed for use by trainees In DSE workshops on 
evaluation of basic education and development training programs: 
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Example XIV. 1 

The model of analysis Involving comparisons between two groups across 
a set of characteristics may be quite simply presented as follows: 



Group A 
N*60 

Group 8 
N r 51 



Characteristics 


1 , 


11 


111 


IV 















Once data has been collated and put In order as shown above t the process 
of analysis ,w111 become easy to undertake, Some similarities and differences 
may be clear even through simple visual Inspection* Some differences may 

9 

have to be tested statistically. (See Chapter XV belt**.) 
Example XIV. Z 

A "before and after" evaluation design may now appear as a model of 
data analysis as follows: 



"Before" 



Kiowkdge 
nutrition 



11-80 



t 



Of Relevant I 

nutrition-related I 
(+) behavior 

— \- — 



"After" 

Ihe Introduction 
of the HFW 
(Nutrition 
Family 
Worker) 



Knowledge of 
nutrition 

N*65 



Relevant 
nutrl 
behavl 
N=61 



ant *1 
tl on- related I 
lor 



Kote that the sample sizes can vary and can be £hown In the model for 
thn reader's Information, 
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Once again* projecting a model (or plan) of data analysis as shown 
above Is helpful In the processing of data and Its adequate analysis to get 
the needed answers. 

Example XIV. 3 

In some evaluation studies, the group A versus group 6» or "before and 
after 1 * analytical designs nay not be applicable. System trapping (or what 1 
have elsewhere called configuration napping or plott'ng) may be needed for 
collating data* for understanding patterns of relationships and for the 
analysis of data. An evaluation study may ask the question; tiny do 
students drop out of a university training course and may conceptualize 
analysis and interpretation of data in terms of the following configurations: 



Cultural/Societal setting 
ft 





Institutional setting 


/ Indlvldu^lX * 




/ non-dropout \ 






\ Individual/ 




\dropoutj/ 






-- l i- 


h l . 



Graduation 



Recrul twent 

The non-dropout and the dropout may be compared with each other In terms 
of Individual characteristics* and In how they might respond-differently to 
institutional and social contexts. The phenomenon of dropping out could 
also be made time-sensitive to identify any phasus and cycles Inherent In 
the process. Without developing such a model of data analysis* the evaluator 
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may not place individual data in context, may fail to ask some exciting 
<trt?st1onS and may not obtain some most useful answers. 

Exampl e XIV, 4 

Finally* a conflgtiratlonal map « a combination of the organizational 
chart and a tjclometrlc netwurk may be used as a model of data analysis 
In cases where peters at different levels of an enterprise are asked the 
same or similar questions on the various aspects of a program* In an , 
exploratory study dealing with the turnover of literacy teachers In a 
literacy program* an evaluator may ask a similar or the same set of questions 
from a wide range of people, such as* chiefs and assistant chiefs, community 
development (CO) assistants, supervisors, community leaders* teachers stilt 
teaching in the program and the teachers who dropped out, and adult learners 
In the program. .The model for data analysis may be drawn as follows: 



Chiefs/ 
Assistant Chiefs 




CD Assistants 




Adults in 
the program 
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Such a configuration*! map ray help both in the organization of data 
(what to say first* what to say next, etc.) and in the analysis of data as 
we iwwe across levels and deal *1th different constituencies In the social 
network* with their special perceptions and interests. 

Before taking leave of this topic* we should indicate that the "set 
of evaluation questions* 1 and the "ftedel of data analysis 4 * are two different 
versions of the same organizing principle. Each should clarify and couple- 
c mni the other. Doth may be used -by the evaluator in the process of data 
analysis. 

Bunnies of intermediate and Final Tables 

As we have suggested earlier, one needs to work back and forth between 
raw data and the needed information as part of the process of data analysis. 
This going back and forth will Involve, as we have suggested above, a 
recollection of the evaluation questions and the development of a model (or 
fun) of data analysis. It will also Involve the design of duonies (or 
, samples) or tables. Sore of these tables will be used for the display of 
data In the final report; but others will be initial and intermediate tables 
which will be used In the processing of raw data but will not appear In any 
report. 

The whole process of data analysis discussed so far can be visualized 
as follows: 
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. Model of 
Data Analysis 



Initial 
Intermediate 
Tabtes 




Final 1 

Tables for — 
Data Display 



Evaluation 
Questions 



Evaluative 
Information 



Processing Raw Data : 
, Coding* Scoring and Standardizi ng 



The first important step In the process of data analysis may often be 



\ 



coding and scoring. 
Coding 

Coding means to assign a particular code to a particular category of 

responsest The following are examples of codes: 

Cote'l Prefers condoms as family planning aids 

Prefers an IVD for his wife 
f Prefers to do family planning by abstinence 

Code 2 Has Insufficient (low) nutrition Information 

Has average (medium) nutrition Information 
lias high degree of nutrition Information 



A 
B 

C 

L 
» 
H 



* Scoring Is assigning numerical values to particutar responses or to 
particular levets of performance. Attltudlnal responses will be often 
qualitative and will need to be scored. The same is true of performance 
scores which may involve observation of performance* Judgment on what is 
observed* and* uttlmately* the change of Judgments into some sort of 
quantitative: scores. 



9 



I 



210 



209 _ 

Standardizing J \\ \\ * 

To standardize scores Is to so treat tii^n that they can be compared 
using the same one yardstick* A profl^of 75 shillings on a 4(H) shilling 
Investment Is not easily comparable with a profit of 15 shillings on a 50 
' shilling Investment, When both profits are standardized as percentages 
(18.75£ versus 30X) they are easily comparable. 

Valuators of development training programs wilt often have to compare 
scores made by Individual trainees on a variety of achievement and perform- 
ance tests. £ach time scores are to be compared* t"he evaluator should 
check If prior standardization of the scores will be necessary. 

In the examples that follow* the processes of coding, scoring and 
standardlzatlnn have been demonstrated: 



" Example XIV. 5 

In evaluating the effectiveness of a training program for agricul- 
tural teachers* a classroom observation schedule used the following Hems: 
TEACHING SKILLS 

(I) Provides introduction Right " Wrong Confusing 
to the lesson 

(II) Changes method Yes Ko ^Reluctantly 
according to need 

(III) Helps students Period!- Hot at Only at the 
recapitulate the caliy all end 
lesson 

(iv) Accepts arid answers Always Hot at Sometimes 
questions ' all 

(v) Gives Individual lo all lo none lo some 
attention poor 

, stuoents 

(vl) Helps the students Always * Hot at b Sometimes 

writes notes all 

fit** a u of the items under the heading "Methods/ In the original schedule* 
nY^-e been used above.) 
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Ve do not wish to make any comments here on the 'merits or demerits of 
the Items as written. The Point we want to make here Is slmpTy that some 
numerical values must be assigned to the judgments made during the observa- 
tion* and that those values must be aggregated for use In data analysis. 
For example, approved behavior may be assigned a score of +1, an Indifferent 
behavior tfay be assigned the value of 0, while an unacceptable behavior 
(which will hinder learning) may be assigned a value of -1- This' will 
enable the evaluator to come up with an aggregated score for the teaching 
skills* as suggested below: 

(1) *1 

(111) +1 

(1v) 0 

(v) 0 

(vl) +1 

Total score: Z 
It Is Injwrtant to note that different types of questions can be 
asked from the same data. For example consider the question: Do student* 
teachers, typically, help children to recapitulate Ideas given In a lesson? 
Looking at answers only, to (ill) above* for all the student- teachers 
tested, an answer to this que ion can be found, 

■E xarole XIV .6 

A questionnaire {filled by each student individually* but sitting as a 
group fn a large hall) sought to evaluate the effectiveness of field attach- 
ment of agricultural students. Questions were asked on a variety of themes* 
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such as; 

(a) Disease control 

(b) Heat hygiene 

(c) Dip management and tick control 

(d) Artificial insemination service 

(e) Veterinary Investigation 

(f) Clinical centers 
, (g) Farm visits 

~T~{h) "Extension and education services 
* (1) New skills learned 
(j) Show and field days 
(k) Management practices 
(I) Unique faming problems 
(oj Individual problems, and 
(n) Field attachment experience. 

Each of these themes* from (a) to (n)» had included a set of questions. 
For example, the theme (a). Disease control* had been broken down into the 
following items: 

I. What notifiable disease^) did you come across? 
. 2. What methods of control and prevention were used? 
Mention the vaccination campaigns you saw. 

4. Enumerate the diseases against which vaccination was done. 

5. What were the reasons for vaccination? 

6. How was the vaccination organized and carried out? 

7. How was the vaccine administered? Indicate any special precautions 



taken. 





8. How many animals were vaccinated? 

9. What was the dosage of the vaccine? 

10,**What was the cost of the vaccine per dose? * 
IK Hon did the farmer pay for It? 

12, What Is the duration of Irammlty for the vy&clnes used? 

13, What was the type of vacd»€ used — live, attenuated or dead? " 

14, What were the problems Encountered In the vaccination campaign? , * 

15, Hon were the~vacdnes used handled? "* * 
As we can see* these questions are a combination of (i) knowledge of 

the student of technical '^formation; (11) recall of "what" was, done and 
"why* 1 In'some problematic sltuatlon'ln the field; (111) Information about 
some local happenings during the period of the student's attachment; And 
(M descriptions of professional actions and technical Practices seen by 
the student during the field attachment but on which the student might have 
had no control, * * t 

In this :ase» the ealuator will first have to separate Hems of 
student's responsibility from those Items which were part of the context; 
and then will have to make Judiiments about the Quality of student performance 
jn the given circumstances , Ihe evaluator may assign A,B»C,0 and F grades 
(or some number grades) to the performance of each student* 

Once, again* we should note that many different uses can be made of 
this data. In addition devaluating student performance, Using the same 
data* one could develop evaluations of dip management or clinical centers In 
the country; learn about the diffusion of new skills within rural communities; 
or learn about farm management practices* In general. 
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Example XIV. 7 , * # 

In an evaluation of the on-the-job performance of cooperatives store- 
men* the^interview schedule to be used with the storemen's supervisors* 
had the following item; 



Excellent Good ' Fairly Poor Very 
Good * Gopd 



BEFORE 

Knowledge and applies * r 

tion Of Co4e of / 
Regulations. * . /' 

AFT£H X 



Notice that data produced by this item will have to be processed Into 
an "improvement* score with a numerical value* before it can be further 
analyzed to study correlations and patterns of Inputs and consequences in a 
training 4 setting. 

The problems of scoring Interview and observation data* to change 
qualitative into some kind of quantitative data, can not be completely 
eliminated in this value-laden world of ours, However* some serious , * 
problems can be Mitigated at the stage of Instrument design and Item 
construction. Tools* and Items Induced in those tools* can be sp designed 
as to elicit answers that are more easily amenable to quantification, 

r Weighting and Combining Scores 

As teacher&we know that in writing achievement tests we can assign 
different narks to questions on the question paper, depending upon £he 
difficulty or the Importance of particular questions. This differential 
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siLrs) Is 



allocation of marks to different questions (and answers) Is called weighting 

Weighting is also Involved In the analysis of opinion and attitude 
questionnaires and observation scheduler Heedless to say that allocation 
^-of-welghts tgjgsp onses on an attltudfnal scale should be Undertaken with 
care* especially , In regard to the values of^etitjal, pos^'ve and negat1_v_e__ 
responses, 



The "Blankets" and Cross breaks 

After the coding* scoring* standardizing and weighting have been 
completed* It Is time for the evaluator to have* a full and complete* overall 
look at -the data. This can be done by developing large-size tabulations 
that show at one glance ttie responses Made by all the different subjects on 
a" total test* a whole questionnaire or some other Instrument. 



Subjects > 

Test 
Hems 



25 



50 



100 



12 34 5*.. 20,.*. 30 



n*1 
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Such tables co^ld include a hundred or more subjects and their responses 
on 30, 40 or 50 Item* on a test* a questionnaire or an Interview schedule* 
These could be big tables and cover half the surface of a walh ^ The partici- 
pants of the OSE workshops In Kenya had started calling these tabulations 

"Blankets*** a name we have retained* 

5 > 
Trafners-e valuators will most often be able to accommodate their data 

-on. one or wore- blankets *- one each for a test* a questionnaire anjd an ^ 

observation scheduler for instance* Where the sample size is in many 

hundreds or even In the thousands* the evaluators will have to use more 

sophisticated methods. Including the help of a computer for data analysis. 

' A careful took at a blanket would suggest many different leads to the 

e valuator in regard to response patterns* and differences and correlations 

between Items* .By consolidating data In -blankets and by- focussing on the 

various rows and columns of the larger blanket, one can develop many useful 

crossbreaks* 

■> Crossbreaks are smaller blankets and can serve useful functions* 
Examine the two crossbreaks reproduced below:^ 



*H. S, Bhola, Eva luatin g funct ional Literacy . Awersham. Bucks, U.K*: 
Uulton Educational PubllcattonsTTS?^ 
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1 










11 , 










!lt 




Question Number 




2 




4 


lotnl 


* > 
1 


2 




4 


total 


1 


2 




4 


total 


"Total possible, 
points & 


5 


5 


5 


5< 


20 


4 


5 


6 


5 


20 


6 


5 




6 


20 


Learner 
A 


5 




2 


1 


11 




2 


2 


1 


S 


4 


4 




5 


1ft 


D 




3 










0 


5 


A 


1* 


T 


5 


3 


4 


17 


C 


4 


'2 


5 


A 


"15 


- 

4 




5 


5 


18 


6 


4 




5 


18 


" D 


5 


3 


I 


2 


11 


2 




2 


1 


" 8 


4 


4 




5 


16 



Fi(iu*F. 1 . Ditto on a mwhttruttfc* test 

■ % 



Leuon, 



-Hi 



FiCUftF, *. AttMifontFand fittat grttdts crossbreak 



In the first cress break above* see under Lesson 1, Items 1 and 2. All 
learners have done almost equally veil on Item 1, and equally poorly on 
Item* 2. UndeV Items 3 and 4 undeV Lesson J no patterns seem to emerge* 
Maybe Items 1 and 2 are not good Items .since they do not help us separate 
good students from bad ones. Or, maybe Hems 3 and 4 are poorly written and 
need to be reworked. 

In the second crossbreak, the changes In performance across lessons 1* 
II and IN can be clearly seen, and different Interpretative questions can 
-be raised through simple visual Inspection. 







Medium 


Low 


tir g h 


Medium 


Low 


tilth 


Medium 


Low 


Leirntr 
A 


II 








8 




* 

* 


\ 
t 


16 


B 




















C 






15 






IS 






It 


D 

* 


n - 
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Data Display bv Tabulation 

Tables to be useful nrj^t be accurately compiled and should be easy to 
read and interpret. 

In the local librae/ or in a bookstore* you may be able to find *o*ne 
annual for writers of term papers* theses and dissertations. These manuals 
provide excellent help on how to compose tables. It Is not witMn the scope 
of this document to provide detailed instructions on how to make tables. 
We will be satisfied with making the following general suggestions; , 
K Number your table* as TABLE ,1; TABLE»2. etc, 

2. Give * title to each table; make the title both accurate and 
complete. 

3. The headings and descriptions used for rows and cMuqhs should also he 
accurate and complete* 

4. Use correct placing and spacing* especially where numbers and decimals 
are involved. 

Do not make up your own abbreviations. Use only standard abbreviations. 
Even when standard abbreviations are used in a table* explain them In the. 
footnotes to the table. 

6i Sometimes, statistics from different years may have to he used in the 
same one table. Indicate which year those statistics belong to* e.g.: 



Population Per capita Radio sets In 

figures In . Income (1974) use (1975} 

mfUlons 



(197S) 



7. Separate "estimates" froto "actual counts." Do not contuse* one with 
the other. 
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0, Wherever necessary* qualify your data. For example* you may havr to 
say: Figures do not include data from Korea* or Domestic workers have not 
been Included* etc. 

9. Standardize your scores, If at all possible. However* If standardized 
scores are misleading* also Include absolute scores. 

10. ' Sometimes* comparative statistics may have to be Included In tables to 
irake sense out of a given set of statistics. One can get a better Mea of 
the level of poverty In a country by seeing, In the same table, the per 
capita Income figures from tl.S.A, or Sweden or even from a richer neighbor- 
ing country. 

pata jHspiavs. jyjjrapfti&s, ■ m 

The question of preparing giiphfcs for displays of data Is important. - 
Graphics communicate Ideas simply and attractively* but they are not 
always easy tn make. There Is a lot to learn ahont making graphics. ]t 
may Interest the readers that there Is a special national Council oji social 
Graphics In the Bureau of Social Science Research In Washington* D.C. which 
recently held a general conference on the topic of "Graphics for Data Analysis 
And Social Reporting. 11 n Is not within the scope of the Present paper to 
discuss the preparation of graphics for data display at any great length. 
Some brief comments are included here on the three most basic graphic displays 
of possible use by trafners-evaluators : (1) the line graph, (2) the har 
graPh and (1) the pie chart, 

The^Hne^orapJ* 

Examine the Hne graph included below. The 0 X axis has been divided 
Into a time scale of .ft centfmfters for each month and. therefore 2.4 centimeters 

O ^ ^ 



£19 



6 * 

*-» ft 
(A O 



\ 

50 
40 



€ «> S io 

as +3 u 



0 t= 



1st 

Quarter 



2nd 

Quarter 



3rd 

Quarter 



Villages with no 
health education 
program 



Villages with an 
ongoing health edu- 
cation program 



4th 

Quarter 



5 th 
Quarter 



-l X 



for a quarter. The 0 Y axis Is divided into parts by .8 centimeters, earh 
part representing ten visits to the clinic. These scales are chosen 
* according to convenience* It is helpful to use graph paper for the initial 
* construction of line graphs (as well as'*of bar graphs). The graphs can 
later be copied or traced to suit various purposes. As can be seen t Hue 
graphs can be used to p resent data'-SOfltewhaLJlramAticallv to mafce_^a -strong — 
Impact. 

The bar graph 

Bar graphs are easy to construct and can jiake effective displays of 
data. Examine the bar graph below. The data about year: of schooling as 
It Interacts with racial origin in the U.S. is presented quite elfeaively. 
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1 



CD 



0-4 57 8 *9-11 12 1315 14 17+ 
Years of Schooling Compfoled » 



The pie chart 

Finally, some data may be presented effectively through a pie chart, 
Examine th*e pie chart below. 
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WorW population. 4,205 mrtfean 




Intern Btoc 

•77% 



Wofjd mconi Ei 7.475 bHIwo dolUrs 
^ ji977| 



34H 




Reproduced from Development and Cooperation , Journal of the German 
Foundation for International Development, Ho. 5^)981 (Cctober/Noveutber) > 
page 7. 

The pie charts are a little More complicated to draw than bar graphs. 
The reason is that we nefe'J to work with proportions of 360, the total 
degrees of the angles projecting front the center of a circle, for example* 
In the upper pie chart: 

1061 : 3144 i* 



1061 X 360 , 314 4 x 360 

T1061 * 3H4T ■ urcrrjra*) 



or 



91 

Similarly, In the lower pie chart 
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zzz 



m$ = 5790 Is 
1685 X &0 , 5 790 X 360 



81 279 



to make pie charts one would also need to have a compass to draw 
circles and a protractor to mark out angles and sectors of the circle* 



Problems of Data" Analysis 
-A". natu ralistic Evaluation 

The problems of data analysis In the context of naturalistic evaluation 

f 

are quite unique. Here* tfoe *eva1 uator would have collected detailed notes 

* { . ? > 

(rather than numbers or responses .that can be quantified) on the basis of 

In-depth Interviews and unstructured observations* tnd must make sense of 

aTl that qualitative data. Tills will Involve content analysis , thematic 

anaJySljS and p r pces s reconstr action. * <• 

Content analysis. According to HoTstl content analysts Is "any 
technllue for making Inferences by objectively and Systematically Identify- 
ing specified characteristics of messages." The aim Is to cateoorlae . The 
categories should be/exclusWe, exhaustive* Internally homogeneous* form an 
Integrated set* be reproducible, credible and must answer the questions the 
eval uator has to deal with. 

IlieiMjtJ[c_anjaJy*l> - As the name suggests* the evalua tor must read 
transcripts of the depth Interviews and observations for the themes that 
pervade the data and then use the jfata to test the hunches and hypotheses 
suggested by the thetnes. 

POTS.^^^ir^i 0 /?- Where change Is the focus of an evaluation* 
the evaluator must try to reconstruct the process by which change could 
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have occurred and the dynamics that might have been Involved. This would 
Involve sympatic "cause hunting." 

In all of the above cases* there will be the need to be tEteoretfcal. 
Some theory, model or conceptualization mist inform our data analysis. In 
the case of overall change* for example, a change model' could be used to 
develop before and after profiles of field realities and to identify the 
intervention that lead to change. 



Discussion of Results 

In the following, sooie general suggestions are made about discussion of 
result's obtained from data analysis:* 



stuoV the precfinditions that prevailed when the training episode of your 
interest begin. The phenomenon of high dropout rates from a college course, 
for example* may be explained better in terms of faulty recruitment methods 
than by what is taught during training. The fuilore of a family planning 
program nay be explained better in t ^ ts °^ tt,e precondition of extremely 
high infant mortality rates in the region. 

2. Potting thinOs in_ti)ntext . Analyze findings in terms of the institu- 
tional and the social contexts of training programs* Do some instf tutiunal 
policies actually go against policies of rehabilitation of distressed 
families or against increasing individual savings? Does tire social context 
promote or inhibit cooperative behavior? 

\\, Sp.Bfiolar "Planning, Change in Education and LtevelopuienM The CtEH 
Hot* 1 In the Context of a Hega Model." View points in Teachin g and Uarnimt. 
Vol, SB, Ho. 4, 1982. J ^ J 



V 



1. Relating with Preconditions and entry .behaviors . As part of the 
discussion of results, reexamine available data on entry behaviors and 
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3. Rel ating with wh5 t js ajready J^own ■ Compare and contrast what your 
data tells you with what Is already known. Do your findings surprise you? 
Are your findings reinforced by what other evaluator* have found In other 
settings? What was expected? What Is unexpected? 

4. Looking for co rre1a_tjojn S T an d causations . Data analysis will typically 
Involve search for correlations and even for cai/sal links, in so doing, 
think of the rival hypothesis « an alternative explanation for what you 
see. Consider all possibilities before making broad assertions. 

5. R^^ir^your ass u^t}or\$ t . It Is Important to keeP on thinking about 
the assumptions on the basis of which the evaluation studfy was designed 

and the evaluation questions were raised. Did those assumptions hold up? * 
How have those assumptions changed. 

6- i jfelgAfoflJ^BJEj 1 * Up? j J^LM-QS?- ?1 <tat ^- Discuss results In terms of the 
limitations of data discovered* as the evaluation design was Implemented 
and evaluation tools and Instruments were actually used. Son* limitations 
of data may Indeed be fatal to the^tudy and to the conclusions drawn from 
It. Another set of limitations may be less severe, but may Introduce the 
need for a high degree of caution In Interpreting results of an evaluation 
study. 

7. Setting up norms for Succ ess and failure . Ihe evaluator must establish 
norms for Success or failure of a program being evaluated. What kinds of 
results will provide the clause for satisfaction? What results will be 
Interpreted as failure? 

,!n dnta analysis one nends to go back and forth between the raw data 
obtained and the evaluative Information that needs to be produced. This 
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will require coding* scoring, standardizing and weighting or raw data, it 
will also require preparation of appropriate tables and graphics Tor the 
display of data. Naturalistic qualitative data requires a kind y of analysis 
different from quantitative data*. In both cases, It will he helpful to have 
a model (or plan) for data analysis to assist In the process of producing 
useful evaluative information* 



* , Thjitjjs to Do or Think About 

1* How ^re evaluation designs different from models (or plans) for data 
analysis? 

* 2. List some problems that you may have come acrossin assigning values to 
* responses an attitudlna) stiles* 

3. What are some questions yoti will Hke to see answered in regard to the 
analysis of naturalistic qualitative data? 
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CHAPTER XV 

BASIC STATISTICS FOR EVAUJATORT. 
OF 0£VEtOP«ENT TRAINING PROGRAMS 

o 

Evaluators of development training programs should be familiar with * 
son* basic statistical concepts and techniques which, they can put to work In 
the analysis of quantitative data. One would, of course* need years of 
rigorous training to become a professional statistician, However, the 
basic essential concepts and techniques of statistics can be learned much 
more quickly. Acquisition of arithmetical skills of high school level Is 
perhaps the only prerequisite for acquiring functional statistical 
lJJLeracy. 

What Is statistics? Statistics Is the science that deals with the 
collection) tabulation and systematic classification of quantitative data 
(and of qualitative data where it can be translated Into quantitative terms). 
There Is desc riptive statistics and there Is Inferential statistics. 
Descriptive statistics helps us, describe In numerical terms entitles* 
proportions* characteristics and conditions that we see * n real life. 
Inferential statistics helps us In using occurrences of events In Inductive 
thinking; and to Infer the general from the specific. 

On the basis of our experience within the context of OSE workshops on 
the evaluatfon of basin education and development training programs* we have 
Identified the following as basic statistical needs of development trainers* 

1. Sunroarlzing the scores of members of a group to understand the 
characteristics and tendencies of the group as a whole. This means learning 
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to determine the roods » median i %an* standard deviaiiuu (ami] variance) of 

i 1 f 

a group of scores* In other wonts learning to measure the central tendency 

j - 

of a group of scores. 

\ - « 

■2- Testing for significant differences between and among the 

performances of different groups, or differences between the same group at 

two different tires* 

J 3. Testing for correlations ■ or for the goinj together of two 

particular characteristics in the same individual or group. (An example 

of correlation wouldkfle the going together of literacy of parents and 

social mobility of their children in the Third World.) 

In the following, we will present the statistical concepts and 

techniques required for fulfilling the needs o£ evaluators of development 

training Programs* We present these techniques as recipes, with emphasis 

on the procedural - steps that ftwst be taken in their application. Nothing 

at all will be said abuut the mathematical logic of the statistical 

formulas chosen for discussion. Those interested in developing deeper 

understandings of these statistical concepts and techniques should refer 

to any standard work on statistics for education and psychology* 

Measures of Central Tendency 

Measures of central tender, are group measures. They help us make 
statements about the characteristics of a group as a whole. These are: 
^1. Node 

2. Median ' * 

J 3. Mean 

4* Standard Deviation (aad Variance) 

9 
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The made Is that score that occurs most frequently In a group of 
* * ■ 

scores. Examine the test scdrcs of a group of ten trainees given below: 

Trainees Scores on. a test on 

knowledge^ nutrition (*) 

1 '80* 

2 * 70 

3 73 

4 72 

5 , 76 <* 

6 60 

7 63 

8 + 72 
g ' 73 

"10 72 

Note that (*) Is the algebraic sign for raw scores. * 
While visual Inspection may help us find the node {also the median or -v 
the mean) of small groups'of scores, ^valuators typically develop a ^ 
frequency table first? with scores ranked In an ascending oi^ a descending 
order: 

Score {*) Frequency (f) 

80 ' 1 

78 1 

73 ^ 2 

72 3 

■70 1 

63 1 

60 1 

An examination of the frequency table above Indicates that 72 Is the 

most frequently occurring score In this group of scores Thus, the ribde of. 



the group Is 72. * ^ 



The frequency table above also tells us that the range of scores Is 

between. 60-80, This can be a useful piece of Information In the process 

i 

data analysts. 

#The median Is a score that cjlvldes a group of scores Into two equal 



r 
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halves. Thus, 50 per cent of the scores In a group are *bove the median 
score and 50 per cent are below it. 

In the frequency table above, we find that our 5th student is opposite 
the score of 72, and that there are three students with that score. This 
is how we proceed In such cases to find out the median. All the three <. 
scores can be seen to be located between 71.5 and 7£rS^ Since there are 
three scores, each of the three can be seen to add an increment of ,33 
(72.5-71.5 divided by 3] to 71.5^as we (nave from 71. £ to 72.5. Me need one 
Increment of .33 to locate the invisible mid-point. Hence the medlart of 
the group of ten scores above is 71.5 + .33" 71.83. 

He should Indicate here that both the trade and the median are sterile 

statistics. There Is nothing imich we can do with them, qfter we have got 

i* 

them. > . . 

• ■ 

The mean (or the, average) of a group of scores, however* provides .a 
useful statistic for measuring the central tendency of groups. Examine 
the following:- 



(1 



(*) 

80 
78 
73 
72 
70 
63 
60 



Mean (X) 



) 



t Total 
713 



f {*) 

80 
70 

m 

216 
70 
63 

J* 



71.3 



r 
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Standard Deviation (and Varia nce) 

Variance Is a statistical measure of ho* Individual scores In a group 
vary from the mean of the group. The algebralostgn for variance Is s * 
where s stands for standard deviation. * 

The mean by Itself may not be an adequate measure of the central 
tendency of a group. One may have to work out the standard deviation (or* 
the variance) of a group to describe v 1t more adequately. 
* Examine the two groups below: 

6rpup_A s Group 6 

V f T *Z h 

60 1 , B5 1 

70 1 S3 . „ > 1 

73 Z 7B & £ 

7Z 3 77 1 

: 70 1 67 Z 

63 1 65 1 

60 1 * 50 '1 

5£ 1 

Hean I. t 71.3 Hean TU * 71.3 

f ^ t 

These two groups have {he exact same mean but even a visual Inspection 

tells us that -kfre two groups are different.- Their modes differ {7Z versus 

67 and 79). Their medians differ (7K&3 versus 67). But most Importantly 

the scores among each group vary differently from each other. Tha^ls, the" 

two groups have different variances {or* standard deviations)* 

* T ■ * 

Ho w to Measure Standard Deviation 

Standard deviation (s) can be calculated by using the following formula: 



n 

"in - 
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Where X stands for the various scores in the group of scores 
n stands for the nurtber of scores, and 
E stands for sutiinatfon. 
Now the standard deviations of the two groups can bt* worked out as 
follows; 
Group A 

EX 2 • 80* + 78 2 + n Z + 73 2 + 72? + U 2 + ?2 2 * 70 2 + 63 2 + 60 2 
i- 6400' + 6034 + 5329 + 5329 + 5164 + bl84 + 5184 + 4900 + 3963 

+ 3600 
= 51,163 

„ (EX) 2 = (80 + 78 + 73 + 73 + 72 + 72 + 72 + 70 + 63 + 60) 2 
» (713) 2 
- 508,309 



s 



' — m — 



^6.233 - 6.0194 

Group B 
for Gi oup 8 

EX 2 • 51803 / 
(h) 2 = 508,369 / 



/ 



* 10,3607 
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The two standard deviations, 6.0194 for Group A and 10. 3607 for GrouP B 
tell us that the two groups are <lu1te a bit different from 2*ch other In 
tui\t variance* 

The standard deviation (s) and variance (s ) are useful statistics and 
can be put to Important uses* 

The t^Je st^for .Jndependtent Samples [ . 

Independent sables are those which are drawn Independently from a 
Population without any pairing or matching or any other relationship 
existing between the (renters of the two groups. (Ideally* these samples 
should be drawn randomly btit It nay not always be possible to do *o In Held 
situations*} 

To see If two Independent samples of trainees are statistically 
different from each other In regard to their performance on a knowledge test 
for example* a t-test can be used. The t-test for Independent samples Is 

t . j/* 2 

/\ n i +n r 2 /V n i V 

%y Is the mean of scores In Group 1 
£g the mean of stores fn Group 2 

Zxj fs the sum of the squared deviations of scores from the mean In 
Group 1 

1*2 Is thf sum of the squared deviations of scores from the mean In 
feroup 2 

nj Is the number of scores In GrouP 1 
n. Is the number of scores In Group 2< 



Where 



and 
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One does not Have to memorize this formula. We only need to know when 
the t-test of Independent samples should be used and where to find it. 

Let us examine the statistics tor two independent samples below: 1 

Group I Group 2 

X, --' 14 ' \ = 10 

z*^ * 106 '* 44 

flj = 15 tijj = *5 

Then, t * 14 - 10 ' 



TiSb * tf~Y7T~T\ 
VTWn) (ft * 15 ) 



* o? s 4 76 

The degrees of freedom for the t-test of independent samples is u } - Z 
which works out to be 28 in this case. Now look at the table of t values in 
the first table at the end of this chapter. Willi £B degrees of freedom* a 
t value of 3.674 Is significant at ,001 level of cunH deuce. In other words* 
with a t-value of 3,674, there is only 1 chance in a thousand that the 
difference perceived between the two groups Is by mere chance and Is not a 
real difference. With our t value of 4,76* we can confidently claim a real 
difference between the two groups. 

The t-Test for Hon independent Sawles 

Uhen two droops of trainees are etched on some Uiaracteristtc or when 
before and after scores of the sape group are being u*ed» we are dealing with 
non independent samples. The formula for finding the t value for matched or 

'This example Is condensed from one given in Donald Ary» Ucy Chester 
Jacobs and Asghar Razavleh, Introduction to Resear ch in Education . Holt. 
197^. 
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before and after groups of scores is: 



t = 



/ rt 2 ^ Mi! 



where D Is the difference between the paired scores of two matched 
individuals or tlm before and after scores of the same one 
Individual 

B Is the mean of the differences (D's) 



tf Is the mmfcer of pairs. 
The degrees of freedom In the case of a t-test for nOnlndependent 
sables Ifc fM. The same table of t values should be consulted to determine 
the statistical significance of the t value obtained,' 

Ajra 1 ysjs of, Jfa£lance: The M est 

if comparisons between more than two grouPs are involved* a statistical 
procedure called the analysis nf variance (AHOVA) and the F-test can be used. 
Readers are referred to a standard text on statistics for a description of the 
ANOVA procedures* 

The Chj -Square Jest J>f. Significance 

Eualuators may sometimes have to compare not means of scores (as In the 
case of t-tests above) hut proportions and f requencies. . The questions may 
take such form: Is there a difference In terms of age for women coming to a 
health clinic? Do men and women differ In regard to their membership In 
cooperative societies? Is there a difference between urban and rural schools 
In terms of their adoption of educational Innovations? Does the size of 



Is the sum of squared difference scores i and 
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landlioldtng make a difference In a fanner's use of new agricultural technology? 
To answer these types of question the eft i -square (x ) test can be used; 

l . E tf o - 

e 

where f 0 is the observed frequency of responses* and 

fg Is the expected frequency - the frequency of responses that 
would have been theoretically expected. If there had been 
no differences between groups or entitles being cornea rid, 



* Let us look at the example below: . 

TESTING FREQUENCY 01STRIBUTI0N OF HIGH ADOPTER AHO LOW AOOPIfcR SCHOOLS IN 
RELATION TO URBAN ANO RURAL ENVIRONMENT USING CM SQUAftE 1 < 





Environment 




Total 




Urban 


Rural 


Frequency 


High Adopter 
Schools 


12 
<B) 


3 

(7) 




Low Adopter 
Schools 


4^ 

w 


11 


15 




16 


14 


30, 



The frequencies. actually observe'* are 12» 4» 3 and 11, The expected 
frequencies have been shown In parentheses. If being urban did not make any 
difference In regard to Innovative behavior* we should have expected high 
adopter and low adopter schools to be equally divided in the urban area and 

*0rest Paul Ochttwa* A study of the Or ganisational Climate of HjgjLiM 
a nd Low Adopter Elementary Schoo1s"ln _ the Province of Saskatchewan , Canada. 
Doctoral Dissertation, Indiana university 71971, 
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we should have obtained the same frequency of responses (■^jjA s 
Similarly. H being rural did not nuke any difference In regard to Innovative 
behavior we should have expected to see the same number of high and low 
adopting schools In the riira 1 areas * 7). 

It should be noted that determining expected frequencies Is not always 
a matter of averaging across categories. Expected frequencies are 
ttaoretjcaljy determined. Suppose there are 30 per cent high school graduates 
among your literacy teachers and the remaining 70 percent have lower than 
high school education. SupPose we ask all our literacy teachers a question 
about their Perceptions In regard to their heeds for further training and 
want to find out if responses differ on the basis of teachers academic 
qualification: hfgh school versus lower than high school. Ihe expected 
frequencies In this case will be Proportionate to the 30:70 division In the 
population. (It will not be a 50:50 division.) 

(low to answer the question If real differences exist between urban and 
rural schools In regard to Innovative behavior, we will aPPly the chl-square 
test: 

z w t IW Z 

X 3 T. t - j ~ 

e 

2 9 Wg)? 4 (wtf 4 (3-7J! ♦ t}±pi = 2 * 2 * 3.3 * 3.3 
» 8.6 

2 

Ihe formula for working out the degrees of freedom In the case of x Is 
(C-1)(R-1}< C Is the number of categories In the column; and R Is the number 
of catoqOrles In the row. Hujs, the degrees of freedom In our case are 
(3-0(3-0 - I. 
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tet us now examine the table on x values. {See Table 2 at the end of 
this chapter*) With 1 degree of freedom we neud a \ value of 6.635 tu 
assert that a difference exists between two sets of responses b with a .01 
level of confidence (that is* there being only 1 chance in a 100 for the 
difference to exist by uere chance and notbe a real difference). Our x 
value is 8.6 which Is even higher than 6.635; and hence wu can assert that 
there 1* a difference between urban and rural schools in regard to their 
Innovativeness* with .01 level of confidence {also expressed asp a < .01). 

Spearman Rank 
Correlation Coefficient 

Correlation means whether two properties or characteristics of 
individuals 90 together. Or* to put it differently nhether two particular 
characteristics covarY. 

Take the example of a class that takes two tests* one for Reading and 
another for Mathematics. The question can be asked: Are students who are 
good In Reading also good In Mathematics and vice versa. Or* are the 
student scores In Reading and Mathematics correlated? 

The formula for determining correlation coefficient is 

p ( rho) . J w 

n(n z -l) 

where d is the difference in the ranks in tno scores {Reading and 

Mathematics) for the same individual and n is thu number of oases 
Involved. 

* 1 
tet us examine the following data. 

'frooiil. S. 8hola* Evalua t ing Functional ti ter acy. /tarshaui* Bucks* 
U.K. : llulton Educational Publications, 1979. 
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Math* 




ftcfttlirtfj 


Rt*diftg 






Student 




Kant 1 


Scwc 


Rank 


d 


d* 


A.B. 


22 


1 


S3 


1 


0 


0 


DO- 


20 


2 


32 


4<S 


2S 


f 2S 


CO, 


17 


J 


4$ 


2 


1-0 


l-W 




IS ' 


4-5 


36 


3 


IS 


2-25 


JUf. 


IS 


4-S 


22 


* 


34 


12-25 


M.IJ< 


14 


0 " 


30 


6 


0 


0 


S-R. 


M 


7 


28 


7 


0 


0 


T,F. 






32 


4<5 


3-S 


I2-2S 


c.k. ■ 


7 


9 


IS 


9 


0 


0 


r.s. 


4 


10 


IS 


10 


0 


0 



2d* - 35-00 



= i 



6(35) 



1 



10(100 - t) 

210 
990 

= 1 - 0-21 

/i » 0-79 



Let us note that correlations can be both positive and negative. They 
can very from -1 to f l. Correlations between 0*00 and 0.30 are considered 
low; those between 0,30 to 0.60 are considered medium; and those? between 
0*60 to 1,00 are considered high, 

tfe must also remeti&er that correlations do not establish cause and 
effect relationships. If scores In Reading and Mathematics are highly 
correlated, ft does not necessarily mean that high Scores In Reading cause 
high scores In Mathematics. 



When more than one of the same scores occur, the midpoint between the 
rank Preceding and the rank following Is assigned to all tied scores. 
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Sua war y 

Valuators must all have a functional level of statistical literacy. 
Willi very little effort on their part* trainers-evaluators can learn tu * 
measure ateans and variances of groups of scores* apply t- tests and x tes>ts 
to test for real differences between the responses made by groups: arid can 
work out correlations between two particular characteristics of, an individual 
or a group* 

Thinas to Do or Think About 

1, Do you have in the institute's library any uf the standard buoks oil 
statistics that could be used by educators and development workers generally? 
If not, it is tine to order some books on statistics for your Institute, 

2. Look at some of the test results already in your files and work out 
qieans and variances for various groups of scores, 

3* if you have some before and after test data for a group of trainees* 
apply to ft the appropriate t-test. * 
4* Are knowledge of nutrition scores always correlated with better 
nutritional habits? Obtain a set of scores for knowledge and for behavior 
and work out the correlation coefficient* 
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CHAPTER m 

U RUING £VALUAT10II REP&RTS 

An evaluation study to be iuost useful uWSt end*in a written report. A 
written report serves at least two purposes. First, it provides an 



the study as well as its limitations. Hot to write a report of an evaluation 



grazing a cow In the pastures all day and not miking it 1m the evenfuyl 
Second* the evaluation report serves as the instrument of cojnuunication 
between and among professional colleagues and others Interested in the s^n* 
or similar problems and issues. 

Evaluation studies have quite often been published; and* sometttt£S* 
{ have brought high professional rewards to evaluators. Ikwt.-er* publication 
and rewards of fame and fortune are not the right expectations to have when 
writing an evaluation report. The si* retards may come, but one should not 
strain to got them everytlihe one sits down to write an evaluation report. 



ItAz much more realistic to think in terms of nuking a few copies of the 
evaluation report to be shared* first and forecast* with pYofessional 
colleagues who should know what your evaluation study has found*^wJ|o can 
discuss your conclusions and suggestion^ with you* and who, perhaps* can use 
the report to improve their performance within the setting of the institution 
to which you ^all^ belong. # 

"""" H 

Ue like to make a distinction here between a tasic professional report 
and overwritten or oral presentations/ The evaluator should prepare one 




study* after data^yatherlng and data collation has been dune* would he like 
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basic and comprehensive report on the evaluation study, If is basic report 
then should be used to makeadlffere'nt written and oral presentation* to 
different groups of (jeople who way t>e interested, among them* policy makers 
and planners* politicians* extension workers* and even fanners and house- 
wWes"who are often the subjects of our developmental efforts, 

The Essential Objective of the Baste fleEort 
and Us Parts ^ 

Ihe essential objective of an evaluation report is to iuake a complete 
record of an evaluation experience 'Including the background and the context 
of the evaluation questions* the assumptions made In posing the queUlOn(s); 
the evaluation 4es1gn and tools used In data collection; the results 
obtained) Conclusions drawn l and practical implications developed frm the 
conclusions of the evaluation study, In otJier words* the evaluation report 
Is a sort of a mirror Image of an evaluatiun proposal as discussed in 
Chapter VI Writing Formal Proposals, for Evaluation Studies/ * 

An evaluation report* however* Is wore than an evaluation proposal 
wrftten ift the past tense. A good c eva1uation report includes all the 
information necessary for a reader to be able to evaluate the evalu ation 
study Itself, That Is* the reader should know exactly what was done and 
'luw; usijig what samples and what questions! and what data was actually 
collected. The reader (oust also be told of the struct the argument 
used In data collation ind analysis* and what conclusions were drawn and *hy, 
In all cases, the reader should thus be able to *>ee the strengths 0 f the 



' Remarks on report writing in this section of the chapter apply mainly 
to evaluation studies that use. the scientific paradigm. 
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study as well as Us limitations; and* where necessary* the reader should 

be able to* do a "secondary analysts" of the data cm his or<her own to draw 

Independent and even alternative conclusions. Ihls means that actual tools 

and Instrument?* and any specimens of stimulus materials used In the study* 

should become part of the report as "Appendices." 

4 

Ihls does not mean, however, that aH raw data should become part of 
the report or should be put In the appendices. A report ts not a device 
for stnrlng and filing all the raw data that was collected for an evaluation 
study. Data Included in the report or 1t| the appendices should be In 
collated form, already organized Into tables and displays* In some cases* 

* ! 

tt may be necessary to prespnt data tn sufficiently "disaggregate^" form so 

that It Is possible for the reader to aggregate data In different .Lays to 

a- 

test assumptions and conclusions of the original evaluaton and* as we 

f 

have suggested above* to draw iltematlve conclusions. 

An evaluation report should typically (but may not always) have the 

parts and sections discussed below; i 

■ * 

The title ^jiatjfe 

Ihe tf tie page of the report should show the title of the evaluation 
study* the nan** of the evaluator(s)* the Institutional affillatl^flj^the 

evaliidtor(s)j and -the date when the report was Issued:. ' ' 

The title given to the report should faithfully reflect the purpose 

o 

and scope of the evaluation stu<(y. This same exact title should then be 
used throughout the stik/ without arbitrary varlatto" - In some cases* It 
may bt *f«1 to have bo^h p a long title and a short title .for the same 
study* Once cno$en*#nese titles should be used in other parts of the report 
without change. The* date of Issup of the report should be shown on the 
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title page* as we have suggested earlier* but SQu.wiiere in the body of the 

report one should also indicate the dates and period of tin*; during whuh 

i 

data was actually collected. (It is possible to ce')ect data in the first 
si* months of 193^ and publish a report in 191M.) 

The abstract ' 

A one- to two-page abstract (that is* of aboot ,600 words) should precede 
tite evaluatio - report* This Should be a complete stannary and must include 
information aboot the evaluation question, samples and procedores used* 
findings and their program indications* A peison who does not read the 
full report should yet get a rairly ^,jo4 idea of contents of the study fruit 
reading this abstract, 

General background 

The first part fn the utain body of the evaluatfun report should be the 
general background of the study. This materia) will no. have to be written 
anew, but should be adapted from the evaluation proposal written earlier. 
Put training for development fn a larger perspective of human resource devel- 
opment for social chanye. Comment on the need for evaluation of training, 
fn general. Be brief. Ho more than a page or two should be utilised. 

focus on your development sector and institution 

focus should then shift to your speefffe development sector such as 
agriculture* cooperatives* health extension* nutrition or family planning, 
and to your institution. TaU about the role of youi j training institution 
and Us contribution to the training of needed m*ntpuuer for development* 
Once, again* brevity is important. One oi two pages or tightly written 
materia) should be enough. 
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Jhe training modej. In use< 

Present the bare-bones of your model of training* Answer questions 
such as: What are the assumptions made about the change Process In the 
training model in use? What are the assumptions made about the change 
agent's rOle7 What are the objectives of training? what are the special 
training methods used? What are the K-A-P {Know ledge- Attitude- 
Performance) claims belnq made In behalf of the training program? (All 
these questions will not have to be answered In each and every evaluation 
report. Hor will these questions be answered In the order In which they 
have been listed here, These are the questions to "think j^lth" as e valuators 
sit down to write their final reports.} 

The evaluation questjon(s) asked 

the evaluation questions as answered In the evaluation study should be 
carefully listed. This list of questions must later be used In the 
collation and analysis of data, fhesc questions should have two linkages, 
On* these questions should relate with the training model-ln-use discussed 
&»1ier* It should he clear how the training model-tn-use generated that 
set of Questions, Two. these uuestlons sho Jd be linked with the subsequent 
organization of data in a latet section and should provide the organizing 

principles for data analysis. 

\ 

Why w<t; this feedback necessary 

This is iti fact a Justification for the choke of particular e 'a 1 nation 
'{U<>st1pns from a whole array of Possible Questions generated by the training 
model and the fn^tltutfonat needs for feedback, fhe ztoterfal from the 

eartlo* evaluation proposal on "justif fcation" and "stgnlf kance" should bp 

iJ 

used for developing thU section. 
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\ Assumptions wade, general to the change and training processes* and 

specific to trie institutional and field settings of yuur evaluation study* 

may be stated here* as relevant. Some of these assuoiptions will have been 

stated In the earlier proposals. Some others may have been urcovered during 
the process of implementation. 

P rocedures And methods usetj 

This section should include the general evaluation design* analyses of 
concepts used and special definitions assigned to terms* indicators used and 
the process of their development and choice* criteria to be used for evalu- 
ating success or failure of tin* program* samples ihusen (and as originally 
intended)* tools and instruments used (which must be placed in the appendix^ 
field work procedures followed, including recruitment and training of 
investigators and time and duration of the field uoric ptase. 

Evaluation design . Go back to the evaluation proposal and reproduce* 
tilth adaptations if any were made* the evaluation desiyn used in the study. 
This rcodel will now be used as a model of data analysis. (Sec the discussion 
on models of data analysis in Chapter XfV of this document.) 

Conceptual analysis and definitions , Yuu may havt undertaken conceptual 
analyses of some concepts such as humanism and self-relume* or may have 
given special definitions of yot,r oun to such words as dropout, literate, 
etc. These should be included in this section. 

Indicators their development and choice. The process used in going 
from larger categories such as self-reliance to subcategories, of larger 
concepts* and, finally • to the choice of indicators which can be wea&uied* 
should be clarified in this section of the evaluation report. 
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Standards and crit eria of success. 1 1 Jfs Important to Indicate In the 
report, the levels of expectations and standards set for Judging success or 
failure of the program being evaluated. The reader should have an Idea 
about whether to be satisfied or dissatisfied with the 30 per cent dropout 
rate from a literacy clas* or the 10 per cent rate of success In the 
rehabilitation of the handicapped. 

^FPlZLP! 1 ) PBifhJrf J£?P°iS£- W'al" sampling procedures. Define 
the samples that wero planned to be used and the samples that were actually 
used. What were the units of response? Was it the housewife* or wa* ft 
anyone else (husband, an older child)* speaking In behalf of the family? 
Was It the chairman of the coranlttee being Interviewed* or was It anyone In 
the comntttOG (o>* more than one person taking turns), speaking In behalf of 
the r.onmtttee? .What was Intended? What actually happened? 

Toots _and instruments. The vartetv of tools and Instruments used 
should be Indicated and their choice justified. Any special procedures 
lined in developing and pre-testing tools should be given. Changes made in 
tools and instruments on the basis of pretesting should be highlighted. 
The tools and instruments should be included in the appendices. 

Meld work. This section should clarify any strategy Implicit in the 
field Wnrk cop (no with dl stances * or with weather conditions^ piggybacking 
on existing system* of transportation? and supervision* etc. It should* 
additionally* include a desciiption of field work procedures and experience. 
Here Investigators used? How wore Ihey trained? How were they supervised? 
How wa* roatniiniratlon hctwrcn the ©valuator and the field investigators 
maintained? Ua* tlmre a *mail pilot study conducted before the final study? 
Did some data have to be coiiected twice? Uhat was the time and duration of 





249 

the field work? Was it found necessary to use a follow up questionnaire 
or interview to supplement the original data? 

Limitations and breakdowns . This section should look backward on 
field work experience ► and forward to the section on data analysts and 
should indicate any breakdown that occurred in held work and any limita- 
tions that became apparent in data collation and analysis later. 

Recording of findings * - 

This section fs the heart of any evaluation report, ft has to present 
a? 1 relevant data in collated fomu in effective displays of ' »bles» charts, 
graphics* list, and vignettes to serve as evidence for all answers given* 
com-nents offered and conclusions drawn. The list of questions drawn up 
earlier and the model of analysis discussed before should be used to 
organize the collation of data, its display* analysts and interPretatlun. 

A separate section any deal with questions not originally asked but 
which, the available data was able to answer. 

Discussion of results 

The findings must be discussed In regard tu the implications fur 
action, and guidelines for future training design. The evaluation results 
obtained must be discussed in terns of expectations* standards and minus. 
These should also be discussed m terois of the strength of data* correlations 
<md Possible causal links. 

Further evaluation and feedback needs 

Karl Pepper has said that our knowledge* and ignuhame increase together! 
A successful evaluation study, by creating new infuraiatiun* tnioht also tell 
us wnat we are ignorant of» or need to know more about, Hew feedback needs 
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should be Identified and suggestions in regard to further evaluation 
studies should he made. t 

Hade a list of hooks, documents and government reports used in the 
implementation of the evaluation study and in writing the report, In case 
of official documents, indicate if they are available to the public and 
where those might be obtained or consulted, 

Appendices 

Hie following kinds of items should go in the appendices: copies of 
took and instruments: specimens and exhibits where appropriate; collated 
data not used in the body of the report but of interest to readers and 
evalu^tors; HsU of nam*** of people* and institutions that cooperated 
wiih the evaluatorfsJ in the conduct of the study* field work schedules* 
maps* etc. 

WH tjji& Repor t ^_ oQatural is.tlc 

Report of a naturalistic evaluation stuo> will tyPicaUy be in the 
form of a Case Study, Some of the ideas discussed above in regard to the 
writing of rePorts of evaluation studies using the scientific Paradigm may 
have some triplications for writing reports of naturalistic evaluation 
studies as well, it mist be understood* however* that case study Is a 
fcPecfa) genre of writing with its own form and style. Unfortunately, 
there art* no rules for wrfting case studies: one learns by reading case 
studies written hy others, and then by writing une's own, 
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While it is not possible to suggest rules for writing case studies- it 
fs possible to malte some fruitful confidents on the general nature of case 
studies, for exaople* ft is useful to coitpare the: writing of case studies 
with creative writing. Creative writing has been described by literary 
critics as the "imaginative reconstruction of reality." By analogy* case 
study writing can be described as the "theoretical recgiisti uction of 
reality* imaginatively done." Thus* case study is the child of both theory 
and imagination* A case study defines, categorizes* conpares, questions- 
hypothesizes, and explains by reconstructing reality in theoretical terras. 
At the saioe time, case study provides "thick descriptions" of environments > 
actors* events* and consequences giving the readei a feeling of involvement 
and a sense of having been there. Thus it is that case study ^ a 
theoretical recvtiSruction of reality* imaginatively done. 

Reports to Uonspeci dlists i 

Written and O ral Repor ts 
As we have suggested earlier* trainers-evaluators should begin with 
writing one basic professional report. This bask report should then be 
used for writing short written and oral reports fur the nonspeciahKt. 
These reports should be written to suit the special mtercsj^ of the 
audience to whom the report is addressed. Oral reports shutild be wade loth 
informative and interesting* using appropriate audio-visual aids. These 
written and oral reports to special audience should in fact become part of 
the process of dissemination of evaluation results* 
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The evaluator must write a basic professional report of t^he evaluation 
study completed. This basic report should then be used to develop a variety 
of written and oral reports to suit the needs and interests of various 
nonspeclalist audiences. The basic professional report of an evaluation 
study can be the mirror image of the proposal of the evaluation study 
developed earlier as part of planning. The report of the naturalistic 
evaluation is typically a case study* It offers special challenges and, at 
its hest f Is the theoretical recons tructTon of reality, imaginatively done. 

Thinks to Oo^or jh ink_ AbquJ: 

1. Examine the report of an evaluation study recently completed at your 
center or at another development training Institute. Oo you find It to be 
a complete and comprehensive report? How would you ojtllne the report to 
make an improvement on tho present version? 

2. Have you read a case study recently? Did you "enjoy" reading It? 
Did yon "learn" Important professional ideas and insights f row the case 
study? 

3* Prepare an oral presentation for a group of farmers based on an 1 
evaluation study done In your country on the subject of agricultural 
Innovation. - ' 



CHAPTER XV II 

POLITICS OF EVALUATJQtl AND EVALUAT I OH STANDARDS 

\ 

We will end this monograph on Evaluating Oey e topiygnt T iajnfng Prpdrjiis 
with a discussion of the politics of evaluation; and a listing of the 
standards of evaluation for an evaluation of the evaluation process itself, 

Ha ndling the Politics of Evaluation 

v Information is power. Information can tie P ut fc o political uses. 
Hence, evaluation* which creates somewhat objective information on the 
effectiveness of development training programs jr on the" i&^actof 
development actions* has political implications* 

How to handle the politics of evaluation? Mo sure-fire formulas t:an be 
tauojit* In any case* most of us who have worked (and survived) within 
bureaucracies are not all that naive about the policies of survival and 
advancement within bureaucracies* Each one of us is perhaps somewhat 
qualified already in the art of "fileiiiansfrtp" and even "one*up*waiwshipP 
Yet* some general suggestions for handling the politics of evaluation may 
be in order, * 

There are two aspects to the politics of evaluation: M the evaluator 
should not get punished for doing the evaluation which may be seen a* having 
produced "embarrassing" information, and (b) the information produced by the 
evaluation study should be put to practical u*e* Political problems do 
arise when* on the one hand* the evaluator seeks to make too much capital 
out of the evaluation study* and* on the other hand, creates information 
that threatens the various stakeholders within the system* Without 
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compromising one's personal and professional Integrity, one can do things* 
however, which will cool the politics surrounding the evaluation study. 

Defend yo ur right to un de r take _eyaj _u a tlon 

Defend your right to conduct the. .evaluation, , (_et People know that 
evaluating is an fntegral Part of good tra Tnlng^ Quote from a presidential 
speech* Planning documents* or from published Prospectuses or reports of the 
training center or Institution* Your institution fs bound to have declared ' 
evaluation to be a necessary part of Its mission* though no one may have 
paid much attention to this particular objective. In an educational 
setting (as distinguished from an administrative setting}* the right to 
evaluate can be da Tended as Part or your research Interest. Youi as a 
'teacher, are supposed to have a research Interest, 

%$cp„ a^Jw prpXU.e 

- There fs need for an evaliiator to keep a low proNle and have a sense 
, of modesty about the evaluation study done. The evaluator should not demand 
to be considered a star In the institutional horizon. The report should be 
Presented without too m"ch fanTare. as a matter-of-fact collection of 
feedback Information on the program. It should not bo touted as a break- 
through or some sort, 

Frainework^oLe^_ctat*ons for evalua ti on resul ts 

No program wfR ever be found to be performing at 100S efficiency level. 
Especially In social change programs* participation levels of 30* may some- 
times br deemed satKTactnry, BeTore Presenting the feedback on Performance 
of a program* one must Indicate what would be a reasonable level of 
expectation of Perfnrm,mct, findings should then be Presented wl thin sudi a 



£55 

framework. In other words* the readers of an evaluation report should be 
provided with standards and yardsticks with which to judge the success or 
failure of a training program or a development action. Without uouiss* 
readers way not know whether to be satisfied or to be dissatisfied with a 
particular set of results* 

As 1 have said elsewhere, the focus should be on finding causes, not 
culprits. This is not to say that the program staff is never at fault and 
that as evaluators weshould be finding alibis for then*. Yet, processes and 
personnel must not be confused in the allocations of credit and blaiue. 
Things must be kept In balance. 

Hegin with a "draft" report 

An important part of the political strategy may be to present the 
evaluation report to colleagues in a "draft" form* offering to do a final 
^ draft on the basis of colleagial discussion and review. In a revision that 
follows, it will be important to neutralize the politics but without 
compromising the Integrity of results, 

■ Indicate Possible actions 

Jndicate the actions that must be taken to make use of the findings of 
the study* Distinguish between things within the Institution's control and 
those outside its control. Start with what the institution can do within Its 
existing mandate such as curriculum revision* preparation of new testing 
procedures* etc* If the implementation of findings demands additional work* 
offer to do it singly, or with the help of a group or a comnlttee. 

What we have suggested above iuy not always work, but It will increase 
the chances of an evaluation study Influencing actions within the setting of 
a training center or a training institute* 
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Evaluation Standards; . ) 

\ 

Evalu ation of Evaluat ions 

*i " ? 
Evaluators should jthemselves be held accountable. Their work must be 

judged according to sow agreed upon standards of technical competence and 

ethics, | 

Ihe Joint Comaftteb on Standards .for Educational Evaluations* of IKS A 

has developed 30 standards which the coranlttee suggests should become the 

working philosophy of evaluators and should guide and govern the evaluation 

efforts of educators (and development workers) * A summary of these standards 

is provided below: 

' Sumnarv ^of^ e Sj.ajrfaj gl s for Evaluation s 
A - jJtjJL ttfl SJ^ujdards / 

Evaluation should serve Practical infonnatton/neeoV 

' / 

1 A 1 Audience Identification 

Audiences Involved In or affected by evaluation should be Identified. 

2 ^ Equator Credibility 

Evaluator should be both trustworthy and competent. 

3 A^ Information Scope and Selection 

The scope and selection of Information collected should enable 
Pertinent questions to be answered, 

4 A fl valuational Interpretation 

Value Judgments used by evaluators should be made clear to readers. 

5 A 5 neport Clarity - 

Objectives* Procedures used* findings* and recommendations should be 

1 The Joint tomnittee on Standards for Educational Evaluation* Sta ndard s 
for Evaluations of Educational Programs , Projec ts and Materials . Mew torfe, 
fi.7."i FicGraw-ffn f Ooofc Co.* T9«l * 
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clearly stated* 

6 Report Dissemination ^ - 

^ Findings must be disseminated for use. 
A 7 Report Timeliness 

Evaluation must be completed on time for use by decision makers, 

8 Ag Equation Impact 

E valuators should encourage follow through by the concerned audiences* 

8, feasibility Standard s ' 

Evaluation should be realistic, prudent, diplomatic and frugal. 

9 B| Practical Procedures 

Procedures should be practical and should avoid disruptions of normal 
work. 

10 B 2 Political Viability : 

E valuators should attract cooperation of various interest groups* 

f 

avoid their attack£> ensure ayainst misuse oF results, 

if 

11 8^ Cost Effectiveness 

Results should Justify resources expended. 

C. Propriety Standards 

Evaluation should be conducted legally and ethkally and should ✓ 

i 

contribute to human welfare. 

U C| ' Fonwl Obligation 

forma) obligations and contracts may be developed between various 
parties Involved .(especially in the case of external evaluotiuMs). 

13 C 2 Conflict of Interest : 

Should be avoided and where unavoidable should be dealt with openly 
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and honestly. 

14 €3 Full and ."rank DJscJosure * f 

Pertinent findings should be fully disclosed; limitations should be 
frankly stated. 

15 C 4 Public's Right to Know 

Public's right to know of evaluation results should be respected 
(unless It Is clearly a nutter of Individual privacy or public 
safety). 

16 ftlgjit of Human Subjects 

flights of human subjects should be respected and protected. 

17 Human Interactions 

* In their Interactions with subjects, evaluators should respect 

dignity and worth of individuals* 
IB Cy Balanced Reporting 

The taportrng should balance both strengths and weaknesses of what Is 

evaluated* 
19 C e Fiscal Responsibility' 

Financial and other resources spent should be accounted for. 

D, Acc^racy_Standards 1 * -* 

Evaluation should convey technically adequate Information, • * 
BlZO D, Object Identification 

7 what Is being evaluated should be clearly identified. 

21 l) 2 Context Analysis * * 

Context of evaluation should be sufficiently described, $0 that its 
Influences on the object evaluated can be Identified 
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"22 Describe Purposes and Procedures 

The purposes and procedures of evaluation should be described in 
enough detaH. 

23 D4 Defensible Information Sources - * 

The sources of information should be described so redder can see If 
those are defensible sources. 

24 0 & 'alid Measurement > 

Evaluation instruments shouid be constructed and applied in Mays to 
ensure validity. 

25 0 6 Reliable Keasureinent - 

evaluation instruments should be constructed and applied tn ways to 
ensure reliability. 

26 Systematic Data Control 

Data should be reviewed and corrected at various stages of the study. 

27 £>g Analysis of Quantitative Information 

Should be appropriate Jnd systematic. 

28 f)g Analysis of Qualitative information 

Should be appropriate and systeiaatic. 

29 Justified Conclusions 

Conclusions should be explicitly justified. 

30 Oj] Objective Reporting 

The reporting, should be*objective and unbiased. 

Some of these standards may seem too tough and sonic too stjue^iish and 
overly fastidious to evaluators working fn cultures other than the United 

States where these standards were developed!, [valuators everywhere shuuld» 

\ 

howevefi take these standards into account to tike extent feasible. 
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Cnnc 1_ud i n£ tonmen ts 

The understanding has now emerged that the <iujiJJM£ of Populations fs an 
important determinant of development* Trainers In development training 
Institutions and centers prepare change age its — literacy workers, adult 
educators, health technicians, nutritionists, cooPerators and agriculture' 
extension workers. These change agents must, tn turn, teach housewives* 
train f aimers and upgrade factory workers to I^Prove the quality of Popula- 
tions within developing societies* Thu&, the work of development trainers Is 
an Important aspect of the overall development effort* 

Indeed, the development training sector Is large fn many development 
countries and significant In all of them. ScoPO for Improvement both fn 
training design and the evaluation of training Is Invnense, Evaluation* 
specially, can bring high economic and social returns. This boot has tried 
to Provide trafners-evaluators h '« the various development sectors some useful 
Heas oth In training design And the evaluation of training* 

lliere are a few Implicit juessages In this took that should be made 
more explicit, iralnlng destdn has become a speciality, but It Is not 
something that only supermen do or can do* Evaluation of the effectiveness 
of development training* agaliu has hecome systematic and technical* but* 
again* It is not something beyjpnd our reach; something which onl> the 

trafoed university research caf^ do. 

\ 

fn basic terms, the Ideas o^f training de&tgn and training evaluation 
are (julte simple. 7hey are not beyond our Intellectual capacity* nor beyond 

otrr personal and Institutional resources- These tasks can be performed now* 

\ 

without much ouutdo help* ^ part of our dally duties* and usfn9 the 
alrea.fy available fnsti hit tonal resources. Heeded is a personal contittmont 

\ 
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to 9Hjw as a professional worker and to do the best pussible job within the 
circuutstances* We challenge our readers to moke Such couuiiitiients* 

Soj^Ddry 

Evaluation produces information which can by used politically. Evalua- 
tors nsust learn to manage the politics of evaluation so that Uity do not 
get punished for "speaking truth to power" i and at the *dme Urn the 
Information they produce is Put to practical uses by decision makers* 
Thirty standards for judging the performance of evaluotors theiitylvti have 
been listed* 

Things to jjo or Think About 

1* Mhat do you think of the practicality of suggestions nude in the first 
pe *hfs chapter for managing the politics of evaluation? 
2. Evaluate a recent evaluation study In terms of thy JO standards for 
evaluation listed above. 
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APPENDIX "A" 

A MNE ml THE ACTUM tRAtNlMC HODEl (AtH) 

The Action Training h^del ( ATN) Is an in-service training matteK it is 
so called because ft demands acjjpn from trainees in the application of skills 
learned durjnq training In their own work on return to their Institutions. 
Skills learned during training must be put to work as part of the dally duties 
back home* without loud And formal announcements of special projects and 
Programs . 

AtH has baen developed under the aegis *he German Foundation for 
international fleveloiWntt Education and Science Branch* as a rosutt of a 
collaboration between the Foundation's Or. Josef Mullor and Professor It. S, 
BfiolA of Indiana University, Bloomfngton, Indiana* a collaboration that 
beqan In 1976. flr. John w\ Ryan* then Director* International Institute for 
Adult Literacy ftethods, t°hran, Iran, and now of the Literacy, Adult Education 
and Rural DPvelnnnmnt fli>parrment of Unesco, Paris* made Important contrfhu- 
tlnns to thp earlier definition of the model as a Participative tralninq 
model. An action oHentatlnn has been added* over the years, v tbt irtler 
participation orientation. ATH Is the result.^ 

Early, betjjnnjngs^, Htonza workshop on lite racy eva ju atjpn, _J976 

Tho first definition of the model . * a participative training model 
emerged within the contort of a two-week J^rkshop on the evaluation of 
literacy programs for middle level literary workers held In Mwanza* Tanzanfa 
during December IQJG- 



ContrlhutiPns of the Kenyan faculty (Joseph Hondo* Tom Hulusa, Alice 
Waka* and Daudf Htwrlbi) In the definition and particularly In the Imple- 
mentation o the Action Training llod^l are *nost QYaiefully acknowledged* 
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T'ic eusphasis H that time was on participative planning So that work* 
Shop organizer* would not be merely creating oppoi tunitieS for themselves to 
deliver lecture on topics they thought were teportaiit* The USk was Instead 
to design a workshop which the local participants considered important, at 
that particular stage In the Implementation of their national literacy 
program. The planning was to be Participative and the organizer^ were to 
be dieting the participants) as equal as others in the group. 

The full owing featores of the Hwanza Workshop should be mentioned by 
way of an initial definition of the training Model as it emerged In Mwanza: 

1. The Mwanza workshop took place within the context of mm formal education 
in Tanzania, It was conducted under the aegis of the department of adult 
education, Since nonformal adult education is a less formalized sector of 
education. It may have provided a context for expei tmahting with different 
training eiethods and strategies. 

2. Tanzania had *n Important on-going program in literacy at the time; 
evaluation of literacy was part of the concern of the government t and those 
who came to attend the workshop were actually responsible for evalodtion 
within the Tanzanlan literacy program. This meant that workshop participants 
knew quite clearly what their needs were. They did not have to Invent 
evaluation problems; and the evaluation problems they did choose had 
relevance to their work. 

3. The Hwanza workshop was designed as a "nationdl" workshop. Tins had 
Mkaoy advantages: it was, comparatively speaking* a low-cost workshop* more 
than a handful of middle level workers could be trained — almost thirty in 
this casei and discussions at the workshop could be rooted within a real 
program and * particular social and political setting. (Tht Mwanza workshop 
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was attended hy two observers from Kenya. On return home* they would help 
the Gnveriment of Kenya decide if Kenya needed and wanted * similar workshop 
on the evaluation of literacy and adult education programs.) 

4. The workshop snuqht to train fiddle- level literacy workers who would 
themselves do most of the Implementation of literacy evaluation as part of 
their day-tn-rtay work. 

5. ft was a two-wnefc workshop bufld around plenary sessions* group work and 
Individual tutoring. Field visits to villages were not Included In the 
program. This omission was deliberate because It was considered unnecessary 
to take rural development workers who spent practically all thefr working 
lives In thf> rural mffou hacv to visit villages to experience field realities! 

6. The workshop was nnt Planned as an "International" event attended by 
vlP's. it was a wnrUnQ worKhnp where participants labored. Ke stayed 
away from hlg horeH. and I ivr»d In a "retreat* 1 setting* away from the city 
and wfthnut a number if cars and buses shuttling around. 

7. Ihp workshop came about at the end of long and systematic preparation. 
The ataSi objectives, Procedures and possible content of the workshop was 
Ihnroughly discussed. A five-page project description was prepared for use 
at a too I of comrajn teat loo hetween and amnnq all concerned with the workshop, 
ft. The wrkshop was based on carefully written Instructional materials. A 
monoilraPh was completed specially for usp In the workshop, namely, 
Evhluatf fig functional literacy hy 11. S. Rhola. (This was later tested In 
nther wortohnps on literacy evaluation and published as a book In 1979 by 
llultnn Educational PnM ka tloijs , Amersham. llucfcs. U.K.)* The monograph was 
arcompanlfd by a dossier of Instruments actually In use withfn literacy 
programs fn various parts of thp world. 
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9. The wost Important aspect of the workshoh, however * was the participative 
c planning of the workshop. The workshop was indeed "invented" lit the local 
setting. There had been a lot of preparation for the workshop* but it was 
general not specific preparation. There were no prior decisions on what 
" specific lectures will be given, by whom and at what time. The workshop 
preparation was In fact preparation fo» a whulc stri of possible workshops* 
only one of which would be required by the participant**, within, the context of 
the local needs of Tanzania at that particular time. Thus* preparation was 
comprehensive and open-ended* the workshop uas anticipated but had tu be € 
re-invented* within the local setting* in participation with the learners. 
It should be mentioned here that the use of j a written monograph for the 
workshop made the task easier. Participants w ere able to understand the 
part -whole relationship between their workshop aid overall subject matter ot 
evaluation. They knew from the written materials what a contpiehensive 
elaboration of a subject or topic would be and how their own specific 
sharply defined needs related to the larger picture. 

ID. An important feature of the participative model was^participation at the 
faculty level. One part of the objectives of the workshop was the cavitation 
of the local faculty through collaboration In teaching.^ 

11. There was considerable emphasis on group work* both for methodulugical 
and practical reasons. The groups provided an opportunity to aH participants 
to discuss and assimilate Important ideas and tu be able to work on topics 
and skilH of special concern to than in their particular work situations. 

12. An innovative use was made of the wall spoce In the lecture hall. The 

'At the tfwanza workshop we were fortunate to have the collaboration of 
Yusuf Kassau and E.P.fL Kbakile both of whom were experienced evaluators. 
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walls were plastered with papor. The learning needs as generated by 
Individual participants In the planning session were written on those sheets. 
These sheets stayed there throughout the two-week Period. The participants 
were able to see what they had wanted to sec done* what had actually been 
accomplished* and what remained to he done. They always had a visual 
picture of the HTe of the workshop before them. ' * 

13. The workshop was steered by the Participants through the mechanism of 
a steering coinnfttee. Every evening without fall, the steering coomlttee 
met to review the experience of the day and oo tuat basis to plan for the 
next day. In addition to the evaluative jnechanlsm of the steering tonmtttee* 
the workshop was evaluated hy sessions* by phases* and* suwnatfvely . at the 
end hy Or, Josef Kuller and feedback Provided to participants. 

14. To relnfnrcc the Participative planning processes* dally programs and 
tW tabled *ere i^ued only at the end of the day — to record what had 
actually happened rather than to have a time tahTe to be strictly followed 
and hy *Mch the bell will ring. This strategy contributed both to the 
processes of formative evaluation and of participative Planning of the 
workshop. 

S e coj>d_u$e_o (JJtc pofjejj 

N^h 3 ** j*°/35 .hpj* °JlJ Ji£E5. cv ova 1 oa tjpn 1 g 77 

Roth the fnrmatlve and srtrmwtlve evaluations of the Hwanza workshop were 
found to he excellent. The ^ame participative model was therefore* followed 
In Mombasa* Kc^ya In 1977, Thft workshop was re-Invented In Mombasa* using 
participative PlannlnQ strategies and wa^, again* partklpatlvely Implemented* 
AM the various mptbodoloolcal features of thc-Mwanza model ^re retained; 
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yet tn Mombasa we had a workshop that was new <md unique to the needs of the 
Kenyan literacy program In 1977. 

Tteiwj the Action Training Model {ATttl 

The participative model was good as far as It went, w> were not sure, 
however, of what happened aftgr the participants loft for their hones. Had 
they been able to acquire enough Information and skills in a two-week work- 
shop to be able to do something with them as they returned to their, posts? 
Had they been able to develop an evaluation proposal on their own and 
foPlement It? Did they get any support within thu system to conduct an 
evaluation study* even If they knew what to do and how? 

To increase the probability that participants will actually conduct 
evaluations In their day-to-day work to improve their Performance, an 
Action Training Model was devised; to participation was added the cotimHuterit 
'lo action, ,The earlier model was augmented by three important features: 

1. Actual development of evaluation proposals* dealing with sous aspect 
of the trainee's own work, was made an essential oart of the curriculum, 
Indeed, all evaluation concepts and skills were learned within the framework 
of preparing an evaluation proposal. 

2, To give time to participants to actually Implement their evaluation 
proposals, a long-term commitment w as made tP them ond* In turn, a luny-tenn 
commitment was demanded from them. The workshops were now organized as 
"sandwiches" - a workshop w as followed by a mid-term panel, which wa!>> 
again, followed by E another workshop. The saute group of Participants w as 
supposed to go through the full cycle of about one year. This was an 
important program comnitipent made by OSE* 
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3. Ihe caPacltatlon of local faculty resources was made more systematic 
and there was a clear concern for institutionalization. For example* an 
Educational Evaluation Resources CotrtnUtee was created In Kenya composed 
local professlnna* workers who would Provide technical assistance to 
Participants in the conduct of their evaluation studies during the one year 
cycle of their enrollment. * 

The_beg1nnjng of the cyde 

Ihe A1H got Its firsf trial in Ker1cho» Kenya In Kay 1979. Nineteen 
participants* trainers front centers and Institutions In different development 
sectors (agriculture* cooperatives * health, nutrition* family planning)* 
attended the workshop. A variety of evaluation Proposals were completed 
during, the workshop and Participants went back to their workplaces with 
Promises to work on their evaluation studies. They were to come back to a 
panel In January T9B0, with evaluation data In hand* to learn what t*> do 
with that data. At the same tlme._an Educational Evaluation Resources 
Committee was established to help Participants with their evaluation studies 
durlni) the months of Hay 1979 to December 1979, 

The experience of the FEflC of working with Participants was not all 
that satisfying and only iu poople Showed up for the January 1980 panel. 
This was something of a shock to the organizers, though It need not have 
been. He know of no other training model In use which demands this type of 
cmmiltmcnt from the Participants of a workshop, Without requests for 
release time* or for additional resources; Participants were supposed to do 
evaluations of their own work* oo their own time — sometimes without support 
f rom the 1 r own Institutions, He AIM workshop does not end after two weeks 
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of yood living and much talkingi one *s asked to make important profess tuna 1 
couriiltments over- a long period of time. 

The organizers learned how difficult couiultiocnt is - <* lo:,g-term 
constituent, without incentives other titan those ot wanting to da d *jood job 
of whatever one) fs doing in one's working life. The organizers alsu 

learned that innovation (such as the ATH) can make the Innovator vulnerable. 

l r . 

The administrator playing the numbers game can a*k: Why aren't there more 
of them? What dirt you do wrong that so t»any participants stayed away? Few 
would look at the other side of the coin add say that 331 of trainees 
did return and that there were six evaluation studies at various stages of 
development. 

Introduction of mixed cycles 

It became quite clear during the Jan^f^ 1980 pAiiel that it was 
unrealistic to expect a 100* nte of^turn from participants. Thirty per 
cent should be acceptable: 405 sho/d be good; anything more than SOX very 
good indeed: and anything above 60X should be considered a miracle. To 
cope with low rates of return* the idea of mixed cycles was invented. Me 
would begin with ZQ people in the first workshop of cycle A, for instance* 
meet m a mid-term panel only those who have shovju satisfactory progress on 
their evaluation studies; and in a second workshop of Cycle A, add new 
participants to bring the number of participants back to 20- The* current 
structure of the ATH looks as follows: 
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Aj...(\ year)...P a ...(S year)...^ 
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B,...(is year). :VP 6 .MV Year)... Bj 

t 



.tt ywh .(■§ year)...c 2 

f 



Technical support Is provided to participants throughout the year by the 
EfRC. as we have already Indicated. t 

A^/ & j j Ajug usL_1 9_D0 jwork shop 

The Aj/Bj workshop took Place In kerftho* Kenya* In August 19B0 and was 
^ attended 6y 20 people — 6 participants from Cycle "A 11 and 14 new participants 
joining the new Cycle "B". / 

The mid-term panel for the second cycle' P^ took place during Karch 
2$-2h> 1981 and was attended by 10 participants* 50* of the Ag/Bj (August BO) 
participants. This was much better than the 22% rate of return of partici- 
pants to the first cyclo panel P fl . 

I The third year cycle began with the ^/C| workshop held during June 1-13* 

* ' f / 

\ 1901. This workshop was attended by 3 participants from the A Cycle, 4 

\ participants from the 8 Cycle end 7 new participants — fourteen participants 

1" alt. (This might seem low as compared, to 19 participants of Aj workshop 

and 20 participants of A^/B^ workshop, but attendance had been kept 

deliberately low to Provide better services to participants attending the 

workshop. I 
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'The mid-tenii panel of C Cycle (P c ) W as cumJuc|ed in February 22-27 ♦ 
1962 and the Cg/Cj workshop is now scheduled for/June 26-July Id 1982. 



Productivity of Workshops / 

The rate for completion of evaluation studies as demonstrated by written 
reports of studies was 371 for Cycle A and 3D.7X fur Cycle 6. However* It 
must be said that only one study out of Cycle A came close to the itual Ity of 
studies produced In Cycle 6, We will be able to judge Lite productivity of 
Cycle C lateV ^ July 1962. 

An important Criterion of Success; 
Kenyans Take Over 

The June- July 1962 (Cg/Dj) workshop is the last workshop which will 
receive outside technical assistance (in the form of Prof. H. S, tihola as 
director of the workshop and Or, Josef Kuller as a mentber of the workshop 
faculty). The rest of the D Cycle and the few cycles to follow will be 
conducted by the Kenyan faculty Independently, with only financial support 
from the German Foundation for International Development. This must be , 
considered a sign of success. 

tn the meantime. DSE hopes to take the Action Training Model elsewherej 

* " * i 

in East. Central and Southern Africa to help where help is requested. . 
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BIBLIOGRAPHY 4 ^ * 

We want to begin with a few comments on the usefulness; of bibliographies 
in workshop documents of this type. Typically* bibliographies are Included' 
In books, monographs and research papers to demonstrate to readers that the 
Piece of writing offered to then Is part of a tradition ef research and 
study. The bibliography shows the choices from the research tradition 
made by the author. Indicating which works the author has considered Important 
and useful and what new departures have been made or suggested. At the same 
tire, a bibliography Is an Invitation to the reader to go Independently to 
some of the books In the bibliography to further cultivate his or her own * 
special Interests. 

Jt Is' the "invitation to the reader" aspect of this bibliography that 
deserves comment here. We realize* of course* that practitioners areV , 
perhaps* not likely to be as enthusiastic readers as professors and 
^researchers are supposed tahe, (There are* of course, avfd readers among < 
practitioners and miserable n on readers among professors and researchers!) 
More to the point, Jt Is unlikely that most of the books In this bibliography 
wllUbe part of the holdings of libraries to which most practitioners In the 
Third World will haVe access. ^ 

We have yet Included this specially developed bibliography In this 
monograph not to show off but Instead to Indicate to the readers that 
training design* evaluation models and techniques* and evaluation of 
development training are important topics which are receiving increasing 
Attention frotn researchers, planners and practitioners alike. It is our hope 
that this select bibliography will also serve a^ a cataJog of Issues and 
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problem of training visigit and training evaluation Uiat are today part of 
the concern of Individual evaluators and developmental Institutions. Lastly, 



ana articles included here and will use their Individual resources or the 
help of their librarian to obtain those materials to be read. 

This bibliography is divided int», three sections. The first section 
deals with the topic of training design . Some very selective books* and 
articles Healing with the various aspects cf instructional and training 
design and training methodologies have been included in tilts section to give 
participants a sense of the "science of training design" that is becoming a 
reality. A second section includes books on i^ juj tjon ng j els jn d tech niques* 
covering both the scientific and the naturalist c paradigms. A third section 
deals with the evaluation of training , especially the evaluation of develop- 
ment training. Readers should note that evaluation needs have cojne to be 
recognized in almost all sectors of development ~- ay culture, cooperatives* 
health* family planning* rehabilitation, media ose and others. 
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it is our hope that our reader* will want to get hold of some of the books 
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GLOSSARY OF TfRHS b 

^^■QJLJL^Jjl^g. gftifeiJ AJjll * A training model developed under the 
aegis of the German foundation for International Development toSE). Bonn* 
Federal Republic of Germany, Ihe model emerged within the context of a 
series of workshops qn the evaluation /*f basic education and development 
training programs- Thc>odfl1 Is so called for Its emphasis on action . 
Trainees are required to make commitments to a full cycle of training 
experiences; first, a workshop where trainees learn generally about 
evaluation and develop specific proposals for evaluation studies; second, a 
old. term panel where the trainees come with evaluation data collected by 
then ^ijrfiwi some s3a months of the Implementation of their evaluation studies* 
review their experience ariu hrePare for daU analysis; and* finally, another 
workshop where old trainees conn* back tn.report on their findings and new 
ones launch upon a new .training rycle under ibc AIM. 

flna 1 Jfsf s^jpf variance ( ANQ VA) , A method of determining whether the 
differences between groups are statistically significant. 

Attrition. Loss of subjects from a chosen sample during the course of 
a study. i 

Aud It^p an £v?1 *!? tjM . F lamination and verification by another 
Independpnt team of the quality of an evaluation plan, the Adequacy with 
which It was Implemented, the accuracy of Its findings, and the validity of 
Its conclusions. 

%*£rJA B £ su r v ?y" An ^^lal survey that can serve as a hase for 
comparing chanqes observed subsequently. 

Bias, A consistent alignment to one particular point of view which 
may make objective evaluation results Improbable. , 

Case study, A detailed description and analysis of a single program, 
ProjpcU course or an tnstructlonal material conducted within its educational 
or snrlal context. 

Code. To convprt a ojven set of data or Items into a set of quantitative 
or qualitative symbols, (Examples: 1, ?, 3 and 4; L, H and 11.) 
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* / 
Coefficient , A statistic (or value) that represents the decree of 

occurrence of a property or relationship. (Example:, correlation coefficient.) 

Concept analysis . The process of "unpacking" concepts to define them 
with such precision that the» will have the same Invariant meanings f u r all 
readers. t> - 

Content analysis . Identifying, categorizing and listing according to 
some rules, ideas; references* feelings or judgments found In a set of 
transcripts* documents* etc. 

Con text. evaluation . Assessing and evaluating, the environmental variables 
of a program. 

Cuntcfrj 91 oum . A gioup whicJi resembles an experimental group (the 
group which is subjected to a particular program or method) as closely as 
possible, but Is not exposed to the program or method whose effect is beifl 
studied, it thus serves the comparative purposes of the evaluator. 

" Correlation . A statistic which Indicates the tfejjree 0/ relationship 

* (going together or happe ning together) between or anung variables. Correla* 

tfons can vary from -KOtf to +1.00. & 

Y 

Cost-benefit analysis ,. An assessment of the inputs and outputs of a * 
program ,1 n terms of their Rtonetary values/ 

» Cost-effectiveness analysis . Ari assessment of the inputs* processes 
and outputs of a program In terms of the effectiveness of means employed 
for the ends obtained. < \ 

Criteri on. A standard by which soioettilnl is judged, \ 

Critcr1on«referenced tests . Tests whose* scoi^s are interpreted according 
to the criteria of performance specifically, defined by tlie teacher in regard « 
to a particular group* and not by reference tu performance of\soine comparable 
populations. * n - (k 

Date . Material gathered during the course of an evaluation, study 
(both Qualitative and qualitative) which Is then used to develop, information 
for dec ts, too making. * ; 
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Data analysis - The Process of identifying Ideas, themes, and hypotheses 
from the data* and the use of data to' demonstrate support for them. / 

"~ Pa ta^ pjeces . Individual tests. Interview schedules* quest loona Ire; a^nd 
diaries that have been completed as part of the data collection phase of an 



^evaluation study. ; ^ 

peftende nt variable A measure {for example, better nutritional habits) 
which 1? supposed to vary as a result of the Introduction of an Independent 
variable {for example, teacMng^Jf^tnj^lona 1 habits by the family health 
educator). s / \ 

Devfon. A model or V clearly established set of procedr/res to determine 
how an evaluatlnn study will, be conducted. (Also see train/no design.) 

Develo pment. The processes that lead to greater Production of wealth 
in a society and a Just ami equitable distribution of such wealth, accompanied 
by Progressive consumption of education *nd culture* and comrot tments _to* 
universal brotherhood . i>eace and preservation of the globe. 

Developffent tra Ining . Training of workers and change agents who will* 
In turn, Impart economic, social and political skills to farmers, workers, 
housewives, and youth to enable them to generate and sustain development within 
their "societies. * 

^ Disseminati on. The process of JJpreaiUng Information abou t_j&vaJ uajjon _ 
objectives and results among those concerned with the evaluation study. The , 
methods of dissemination may be written or oral. 

E va luation. Objective and systematic collection of Information about a 
program* project, or Instructional material for Its Improvement. (More 
recently In literature* evaluation Is betmi defined as the "s/stematlc 
Investigation of the worth or merlin of an object; e.g., a program* project* or 
Instructional materfal.) 

Ej^ajua t loj^s^Uem. An arrangement of methods. Procedures ami Plans of 
action designed to provide decisionmakers with Information on the Inputs, 
output*, context And process of a given program. 

^9!T$).w?}u?tiM- ^valuation conducted by evaluators not on the 
staff of a program or Project. ^ 
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Extrapolate . 1o infer from what Is known* something tliat is unknown. 
(Population figures for a country for the year 2000, may be extrapolated » 
from the population growth figures dut <ng 1950-60, ) V i 

4 

feedback . A term borrowed from electronics: the return of part of the 
output of a system into the Input for purposes of modification and control 
of the output, tn the context of program planning* feedback means evaluative 
Information on program effects. 

Field test . A preliminary study of a program* project or Instructional 
material In a setting very similar to the one In winch it will, be later t 
Implemented of used on a much larger scale. 

formacive evaluation , Evaluation conducted during the very formation 
of a program* project or Instructional material* ^ 

CeneraUzabillt^ . Ihe extent to which claims and assertions made about 
a program project or instructional material in one setting can be applied In 
other settings, / £ 

Goal* free evaluation . Evaluation uf outcomes of programs and projects 
where thk evaluator functions, wtthuut knowledge of the puiposes and tjoats uf 
^a program or project. - 

ihuLn resource development (HfiP) . The education and training of 
taanpowefr * tnHh for formal and informal sectors ot the economy, using butli 
format [and fEbnforral systems of instruction. 



lrittependent variable , A treatment variable Introduced In an, evaluation 
,a new teaching method)* anticipated to create varying 
dependent variable (for example, performance on a te^ti). 

Indicator . Something that Indicates* points* signifies; a gaoge that 
represents another entity. Thus, a high drop-out rate in an adult education 



setting! (example; 
effects] on a deper 

Intii 



programj titay &e a° indicator of a lack^of community motivation. 



in Jut evaloation . Assessing tlie various* resources u*ed in conducting a 



program 
In 



arrangeuents or systems fur the implemoittdtluii of programs oi projects on a 



tltotion building. The process of developing organisational 
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long-term hash. (To institutionalize is to make a Program more or less 
permanent through Institution building.) 0 

Instr ume nt?. An assessment device (test* questionnaire* Interview 
schedule* or observation schedule) used Tor the purposes or evaluation. 

Internal eyal waMon. An evaluation conducted by a staff member from 
within the .organization whose proqram* project or Instructional material Is 
.being evaluated. 

Le vel of jj&njjjffflce. /j Predetermined probability value which Is^ed 
to decide whether the results fof an evaluation study occurred by, chance anfr 
were nnt really a consequence V a Program* Project or Instructional material. 
j(i ; ^ .01 m**n? th*t thrre ufjhn probability of osily er.e In s hundred fa** 
the Program effect to have appeared by chance,) 

►Ha^A5etflent_ P.nn3tJon_ sysJemjHyjSJk A System (computerized ♦ manual or 
a mix of the compti tori zed aod manual) Including planning and Implement a fter* 
data 1" regard to a Program or project, (See also Monitoring.) 

Mat^hijig, The Process by which subjects assigned to different groups 
are made to be^as equivalent as Possible. (Matching may be done on such 
variable*; as sex* a*Je, education* socioeconomic status* etc. A set of twins 
would he Perfectly matched for purposes of some studies.) j_ 

Hea'n. The sun of a group of scores divided by the number of Scores/ 

Hedijfj. The score In a group of scores that Is midwa 1 ' in thj! distribution, 

Mode. The scorr In a group of scores that occurs most ofteQ* 

Model. A desiqn* description or analogy used tn helP visualize or make 
understandable something thai is more complex. 

!fei!Si-?i!£!2S^-L5P? tysJh* ^ procedure similar to detective work wherehy 
causes and effects are hypothesized* tested and analyzed to arrive at the 
most UVeljr pattern? of events and their consequeojees . 

t HojHtorlng, fo monitor Is to check on an on-going program or project 
for' flaw^ or breakdowns * to enable decision-makers to regulate activities and 
tcj undertake_correct1v(? action- Mon ,f or1ng Is typlcallv based in a management 
Information sVstem, 
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Naturalistic: inquiry Paradi oa. Study of behavioral phenomena In natural 
settings and in their normal context* using methods drawn from ethnography* 
anthropology and sociological field studies. Also called the ethnogi aphic or 
the phenomenologtcal paradfgnh 

Heeds assessment . The process of ascertaining the learning needs* 
health needs or other developmental needs of henef ici aries of educational 
and development: programs* Needs assessments are a mix of *felt" needs 
expressed by beneficiaries and new needs "fashioned" by change agents-* 

No significant difference (nsdl . A label which is used to say that the 
observed difference between two statistics cuuld have occurred by chance. 
(See Level of confidence above*) 



Hon formal education . A collection of organized or semi-organized 
educational activities, operating outside the- formal education system and 
meeting the immediate educational needs of both conventional and non- 
conventional learners* (Foraal education is that which is provided by. 
schools, colleges and universities. Informal education is that where neither 
the educator nor the one being educated are conscious of the prucess of 
tea.hlng-learnliig taking place,) 

yorm. A value or pattern* of values' representing the typical pei formante 
of a gr.oup or population, , 

• Norm- referenced tests . See standardized tests; 

Objedves-ireferen ed tests . Tests, whose scores are interpreted 
according to the objectives which a program project or course was designed 
to*teach» without comparing performance of other groups on the test. 

Operational seminar , A training method developed within Unesco wherein 
participants experience on a reduced time-scale the total process of 
community work* problem diagnosis, needs assessment, Meld organisation* 
materials design and evaluation in an actual field setting. 

Output evaloation . Assessing the quality and quantity of the final 
product Is) of the program, also taking into account any unintended ^products 
of the program. 
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Paradigm. An exanole or pattern; a very clear example of an archetype. 



In evaluation, a par a diem is equivalent to the 'Intellectual Ideology 1 of an 
evaluator, " 

Parameter. Any one of a set of properties whose value determines the 
characteristics or behavior of something, 

^I^}S^^^^^!PS 0 ^^JL' Designs* Procedures and methods of planning, 
Implementation and evaluation that ore built upon the active Involvement of 
the would-be beneficiaries of programs and projects. 

Pppulatio-t. All the persons* In the group to which the results from a 
study will apply. (E.g., "all cotton farmers In the lake Regions of Tanzania, 
all women tn the child-bearing age In Indiana.) 

Post-J:est. A test to determine the effects of a program, project or 
Instructional material after application or completfoo. 

Pre-test* A test to determine level of performance before the start or 
application of a program, project or Instructional material. 

Prod lew ^comptex. A whole set of interrelated Problems (of planning, or 
of management, or of evaluation), emerging around a decision point within a 
system. 

Prpc_ess_eva1uaUPil- Assessing procedural strategies and comparing 
effectiveness of different approaches to Instruction, extension, animation 
and organization. 

Product evaluation. Assessing the effectiveness of currlcular or 
Instructional products. 

Qy3UJ?ilH. d?*?- Fa ^ts, claims anjf assertions In narrative form, and 
not In numbers. (Qualitative dtfta can, however, be converted into numerical 
form by coding and scoring.) 

Qu^ntUatJj*e_data. Farts, claims and assertions presented tn numerical 
forms. 

Qujc^jippraisajs. Quick evaluations, less comprehensive and less 
exhaustive than regular evaluations, conducted under conditions of emergency 
to Investigate the cause of a breakdown, to anticipate problems or to get 
early returns on the Impact of a program. 
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Random sample . A representative portion chusen from among tne population 
each individual in the population has *n equal chance of being selected each 
time a selection Is made. 



ReVjabiljty^ . The property of an instrument giving the same reading or 
score Hhen used by different investigators on the saute entity* or by the 
same investigator repeatedly on the same one entity. 

flepllcatlon . The repeat of an evaluation study with all essential 
aspects of the study remaining unchanged. 

Responsive evaluation . Evaluation that responds to the information 
needs of the various stakeholders In a program by providing evaluation feed- 
back on concerns and issues raised by them, rather than evaluating what the 
evaluator thinks is worth evaluating. 

Sample . A part of a population chosen according to some method to 
represent the total population. 

Scientific inquiry paradigm . The approach borrowed from the hard 
sciences involving experimental design, randomized samples, controlled 
groups and statistical analysis. 

Situation-specific strategy (3-S) model of evaluat ion. A five-step 
model that relates evaluation with change* requires the articulation of means 
and ends within an educational or a developmental program* proposes the 
development of profiles of information needs* Suggests that situation- 
specific and strategic agendas for evaluation be developed* and that the 
choice of evaluation methodologies a ndr techniques be both technically 
appropriate and practically feasible witftni the setting of evaluation. 

Standard deviation fs) . A measure of variability calculated on the 

basis of differences of individual scores within a group from the group mean. 
2 

(s Is called* variance.) 

Standardized tests . Tests Whose scores are interpreted in comparison 
with some norms established in terms of some larger groups or populations. 

Statistic . A summary number that describes the characteristic or 
property of a sample. 
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St atistical analysts- An examination of complex relationships between 
variables using empirical data and rules of statistics. * 

Statistics. The science of methods for analyzing data obtained from 
empirical observations to make descriptions or Inferences. Thus« there Is 
descriptive statistics, and there Is Inferential statistics. 

SjyF!!Sii^^yALH?.t * on ■ Assessment of the Impact of the total product* 
program* etc.* comparing observed effects with anticipated or desired effects. 

System. A whole emerging from an Interacting and Interdependent set of 
Parts* subject to a corrmon plan and having a common Purpose . 

SystOTS^ nwdel ■ A model that looks at social reality as a system that 
can always be described In terms of Inputs* processes* outputs and context. 
(See also model and system.) 

Taxonomy. An orderly classification that has some theoretical under- 
pinnings. 

IIM *jL d e sc H j> 1 1 on . Detailed and faithful descriptions In the form of 
photographic records and protocols or 1 written case studies. 

" i^I^S 1 !- A model or a clearly established set of procedures to 

develop a training program. Involving planful selection of educational 
objectives, learner characteristics* teaching methodologies and learning 
environments. 

Id£ojg_ u Jat1on. Comparing and testing results from two orwnore different 
approaches to the solution of the same one problem. 

'Unit of anal ysis. The social unit such as Individual* husband-wife 
dyad* family, group* organization or community which Is the focus of 
^Interest for the evaluator; which will determine the*organtzat1on of data; 
and about whose behavior statements* claims and assertions will be made. 

Unobtrusive measures. Methods of examination In which the evaluators 
do not materially Interfere In the situation, but rely on Indirect procedures 
to gather data. 

Validity, The ProPnrty of an Instrument to be able to measure what It 
was supposed to measure, ° 
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Variable, A characteristic that, can take on different values. 

Variance . Admeasure of variability calculated on the basis of 
differences of Individual scores within t group from the group wean, 
square Foot of variance givfts the value of standard deviation ts). 
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